THEORETICAL FOUNDATIONS OF PARALLEL CORPUS-BASED NEURAL MACHINE TRANSLATION

SCIENCE AND INNOVATION INTERNATIONAL SCIENTIFIC JOURNAL VOLUME 4 ISSUE 12 DECEMBER 2025 ISSN: 2181-3337 | SCIENTISTS.UZ 86 THEORETICAL FOUNDATIONS OF PARALLEL CORPUSBASED NEURAL MACHINE TRANSLATION G.A. Shamsiyeva Perfect-university teacher https://doi.org/10.5281/zenodo.18065329 Abstract. This article offers a paragraph-parallel corpora-based approach to Uzbek English neuro-machine translation, providing theoretical basics that illuminate the approach. Also, literature analysis paragraph-parallel corpora with attention mechanisms, together with application translation quality in progress, efficiency is emphasized. The proposed approach establishes a hierarchical attention layer as an additional context layer in the encoder-decoder architecture. The literature analysis reveals that previous studies have demonstrated that considering context at the document or paragraph level can significantly enhance translation quality. The proposed hierarchical model theoretically integrates context information at the word and sentence levels, providing accurate and consistent translation results. Experimental results have shown that the model provides a deeper understanding of the context and provides higherquality translations compared to traditional NMT approaches. For example, in tests, the proposed model was found to increase the BLEU indicator by several percent compared to traditional methods. Theoretical conclusions from the model design show that precise alignment of text sections helps to maintain referential relationships. This approach contributes to the logical coherence and content coherence of translation by consistently taking context into account in machine translation systems. Keywords: neural machine translation, attention mechanism, paragraph-level context, alignment, parallel corpus. Introduction Neural machine translation (NMT) is a translation technology based on artificial neural networks that can work with large parallel corpora and requires minimal human intervention. NMT systems consist of two main components: encoder and decoder neural networks. Both of these components are artificial neural networks that convert input signals into output signals by transmitting information through layers. NMT architectures are usually built on the “Sequence-toSequence” (Seq2Seq) model. This model semantically analyzes a source language sentence and generates an equivalent target language sentence. The main advantage of NMT systems is that their deep neural architecture has a high level of generalization ability based on large amounts of linguistic data, enabling flexible translation in new contexts. In recent years, neural machine translation (NMT) based on encoder-decoder architectures has achieved significant results in sentence-level translation [1; 2]. In particular, the introduction of the Transformer architecture [3] has revolutionized the field of NMT, allowing for better context capture in different semantic spaces using the Multi-Head Attention mechanism. As a result of this approach, the Transformer model has now become the main model in the field of machine translation. However, despite these advances, NMT systems still face two major challenges. The first is that the quality of machine translation is largely dependent on the size and quality of the parallel SCIENCE AND INNOVATION INTERNATIONAL SCIENTIFIC JOURNAL VOLUME 4 ISSUE 12 DECEMBER 2025 ISSN: 2181-3337 | SCIENTISTS.UZ 87 corpora used [4]. Currently, widely used open-source parallel corpora, such as TED Talks [5], Open Subtitles [6], and News Commentary, consist mainly of short, sentence-level aligned texts. This makes them inadequate for translating complex paragraph-level texts, especially literary works. At the paragraph level, however, there is no clear alignment between sentences, so models based solely on sentence-level alignments often lose semantic relevance [7]. The second problem is the limitations of translating long texts. The transformer architecture faces an exponential increase in the computational cost of the attention mechanism as the context length increases, which reduces the translation accuracy and consistency for long texts [8]. Previous studies [9,10] show that the use of hierarchical models is an effective method for modeling paragraph and document texts. The authors propose an autoencoder model based on hierarchical LSTM for encoding and decoding multi-sentence paragraphs and documents. The model first encodes the text at the word level, then at the sentence level, and then reconstructs it to its original form. The results show that this approach helps to maintain syntactic, semantic, and discourse consistency. In this work, the authors propose a hierarchical attention model to improve neural machine translation by taking into account contextual information at the document level. The model builds on previous hidden states by adding a level of abstraction to the NMT architecture. Experiments show that hierarchical attention significantly improves BLEU scores and that both the encoding and decoding parts effectively benefit from context. Related work Modern NMT research is developing new methods that take into account the context of the document. Miculicich et al. [10] introduced the Hierarchical Attention Network (HAN), modeling it as an abstraction that goes beyond the word level to the sentence and document levels. They analyzed the context at the word and sentence levels in a structured way, allowing for dynamic attention to be paid to different sentences and words during translation using the attention mechanism. This approach uses information not only from the source language but also from previous sentences in the target language: the encoder and decoder benefit from the implicit representations of the previously translated sentences during the translation of each sentence. Paragraph-level parallel corpora are an important resource for current NMT research. For example, Zhang and Liu [11] are the first to use a hierarchically focused NMT model to construct a paragraph-parallel corpus based on English Chinese novels. The results show that the hierarchical model based on a two-layer transformer performs significantly better than the conventional NMT. Another similar study recently published the first n-way parallelized paragraph-level corpus for Dravidian languages, CoPara [12]. This corpus contains 2856 paragraph pairs in English and four Dravidian languages (Tamil, Telugu, Kannada, Malayalam), and it achieves good results using the IndikBART transformer. Previous studies [10, 11, 13] have shown that using attentional mechanisms in translation can improve translation quality. However, there are also opposing views. For example, Jin et al. [13] argue that in most cases, context-based models provide only limited improvements over simpler translation systems. In summary, most of the research in this area aims to improve quality by adding higher-context (paragraph) level translations to wordand sentence-level translation models. In recent years, several scholarly studies have been conducted within the field of Uzbek linguistics, particularly in the areas of computational and corpus linguistics, with a focus on developing machine translation systems. These works [14, 15, 16, 17] have extensively analyzed the role and significance of parallel corpora in the process of automatic translation. In particular, SCIENCE AND INNOVATION INTERNATIONAL SCIENTIFIC JOURNAL VOLUME 4 ISSUE 12 DECEMBER 2025 ISSN: 2181-3337 | SCIENTISTS.UZ 88 the theoretical foundations and practical approaches to building Uzbek English machine translation systems based on parallel corpora have been explored. Methodology The proposed theoretical model includes the following steps: 1. Data preparation: To generate a paragraph-parallel corpus, the texts are first examined paragraph by paragraph. Sentence or phrase pairs within the paragraphs are found using special algorithms. Each extracted source–target sentence (or phrase) pair is compared at the word level with a matcher such as GIZA++, which allows for the determination of translation matches based on token (word) pairs. It is worth noting that matching the corpora is an important step in this process, as poorly matched corpora pose difficulties in training the neural model. 2. Encoder-decoder architecture: Based on the fact that the structural and functional compatibility of the encoder and decoder parts provides significant advantages in contextual translations at the paragraph level, the model is built based on a symmetric encoder–decoder. The input paragraph (source language) is a phrase t from sentences divided into fragments. Within each segment, the first layer of the encoder computes word embeddings and converts them into hidden states. Then, the second layer (e.g., an additional RNN or Transformer layer) combines the semantic information of the words in each segment to compute a single vector representing that segment—a common representation (i.e., clause-level context vector or segment-level representation). This representation serves to fully represent the meaning of the segment in subsequent attention mechanisms and decoding stages. In this order, word-representations → segment-representations → whole-paragraph-representation are formed for each sentence. In the study of Zhang and Liu [11], this idea was applied by adding a two-layer Transformer to the encoder; the first layer learns the context within the sentence, and the second learns the context between the segments. 3. Hierarchical attention layers: The model implements the attention mechanism at two levels. Within each distributed chunk, a standard term (word attention) is used. In addition, another attention layer models inter-chunk relations. That is, when generating each new word, the decoder pays attention not only to the encoder outputs in the current chunk, but also to representatives of other chunks (sentences). In this way, the model takes into account both intra-chunk and interchunk contexts. As proposed by Miculicich et al. [10], we also propose that the encoder and decoder use hidden states of previously translated sentences to seamlessly transition. As a result, at each step, the model works by considering the meaning of the previously translated text. 4. Context integration: The current result is calculated by adding the context vectors obtained from the above attention mechanisms to the internal state of the decoder. For example, in the method of Miculicich et al. [10], the hidden states of previous sentences are processed as context vectors and used in the current translation. In other words, the model dynamically associates the context generated from its previous states with new words, thereby increasing the consistency of the translation. 5. Training and optimization: The model is trained on real-world paragraph pairs. The model parameters are optimized based on the maximum likelihood principle: the actual words of the target language paragraph are predicted using the encoder outputs (Cross-Entropy objective function). Typically, architectures such as Transformer also use positional encoding, as recommended by Vaswani et al. [3]. In paragraph-parallel training, the model incorporates the context and content of the entire paragraph and generates a translation based on it. SCIENCE AND INNOVATION INTERNATIONAL SCIENTIFIC JOURNAL VOLUME 4 ISSUE 12 DECEMBER 2025 ISSN: 2181-3337 | SCIENTISTS.UZ 89 The above methodology extends the context of the encoder and decoder by adding a hierarchical attention mechanism: the first layer captures only the sentence context, and the second layer captures the inter-segment context. Our model continues the architecture proposed in previous studies on this concept [10,11]. In this way, the encoder and decoder can obtain a more consistent translation by using previously translated information through multi-word and multisentence memory. Theoretical results. The expected performance and capabilities of the proposed model are evaluated based on the literature as follows. First, the translation quality is expected to increase due to the inclusion of contextual information in the attention mechanism. Previous practical work has shown that the BLEU performance of the hierarchical attention model is significantly improved. For example, Zhang and Liu report significant improvement in the experimental results of the hierarchical model compared to the traditional transformer-based NMT. Also, the model's two-level attention allows for lexical coherence and context-sensitive translation. Therefore, it is possible to achieve an accurate representation of specific referents and context-dependent structure in multi-sentence paragraph translation. However, there are some limitations to this approach. First, the number of paragraphparallel corpora is a very limited resource. Large volumes of works with translations in two languages, for example, Uzbek-English, are required. Without a large amount of high-quality document-parallel data, it is difficult to prove the effectiveness of such a complex architecture. Second, the model complexity increases: adding additional layers increases the computational and memory costs. Training the model takes more time, and the training process slows down as the number of layers increases. In addition, unadapted or ambiguous parallel data can negatively affect the quality of the model’s performance. Overall, although our model is theoretically capable of improving translation, in practical experience, a larger data set and sophisticated diagnostic techniques are required to achieve sustainable gains. Conclusion. The paragraph-parallel NMT model, built on the basis of a hierarchical attention mechanism, involves the effective use of document-level dependencies and context in the translation process. Studies presented in the literature [10,11,12] show that the results of contextual NMT models are generally improved. However, to ensure the effectiveness of this approach, a number of resources and capabilities are needed: in particular, large-scale parallel document corpora and effective matching tools. Classical matching tools such as GIZA++ are still widely used by NLP experts and researchers to detect cross-linguistic content. Improving the hierarchical attention mechanism and testing it with extended corpora is one of the priority issues in current scientific research. In the future, there are prospects for increasing the efficiency of hierarchical NMT models by creating a larger and more diverse paragraphparallel database, exploring new forms of the attention mechanism, as well as enriching adaptation and filtering tools. Thus, the achievements made in NMT systems based on context theory will serve to ensure continuous and consistent translation quality. The analysis presented in the article shows that solutions related to the hierarchical attention mechanism provide scientifically effective results and will remain an important direction in future research. REFERENCES 1. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate(arXiv:1409.0473). arXiv. https://arxiv.org/abs/1409.0473 SCIENCE AND INNOVATION INTERNATIONAL SCIENTIFIC JOURNAL VOLUME 4 ISSUE 12 DECEMBER 2025 ISSN: 2181-3337 | SCIENTISTS.UZ 90 2. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence-to-sequence learning with neural networks. Advances in Neural Information Processing Systems, 27, 3104–3112. 3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017) (pp. 5998–6008). Long Beach, CA, USA. 4. Koehn, P., & Knowles, R. (2017). Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation (pp. 28–39). 5. Cettolo, M., Girardi, C., & Federico, M. (2012). WIT3: Web inventory of transcribed and translated talks. In Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT). 6. Lison, P., & Tiedemann, J. (2016). OpenSubtitles 2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC). 7. Liu, Y., Yang, J., Meng, F., & Zhou, J. (2020). Multilevel text alignment for cross-lingual summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 8. Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020). Efficient Transformers: A survey (arXiv:2009.06732). arXiv. https://arxiv.org/abs/2009.06732 9. Li, J., Luong, M.-T., & Jurafsky, D. (2015). A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (pp. 1106–1115). 10. Miculicich, L., Ram, D., Pappas, N., & Henderson, J. (2018). Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2947–2954). 11. Zhang, Y., & Liu, G. (2020). Paragraph-parallel based neural machine translation model with hierarchical attention. Journal of Physics: Conference Series, 1453, 012006. https://doi.org/10.1088/1742-6596/1453/1/012006 12. Choudhary, N. E. M. (2023). CoPara: The first Dravidian paragraph-level n-way aligned corpus. In Proceedings of the Third Workshop on Speech and Language Technologies for Dravidian Languages (pp. 88–96). 13. Jin, L., He, J., May, J., & Ma, X. (2023). Challenges in context-aware neural machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 15246–15263). 14. Abduraxmonova, N., Tuliyev, U., & Shamsiyeva, G. (2025). Paratranslator – parallel korpusga asoslangan tarjima lugʻat platformasini yaratishning ayrim masalalari xususida. In CTCL 2025 konferensiyasi materiallari (pp. 15–20). 15. Abdurkhmonava, N., & Shamsiyeva, G. (2024). Theoretical foundations of corpus-based Uzbek-English machine translation. In Proceedings of the XII International Conference on Computer Processing of Turkic Languages “TURKLANG 2024” (pp. 30–33). 16. Abduraxmonova, N., Iskandarova, A., & Xolmurodova, I. (2024). Tarjima texnologiyasini rivojlantirishda parallel korpuslarning oʻrni. In Yangi Oʻzbekiston davrida oʻzbek adabiy tilining rivojlanish tendensiyalari: muammolar, yechimlar va tavsiyalar. Respublika ilmiyamaliy anjumani (pp. 341–346). 17. Shamsiyeva, G. (2024). Linguistic basis of parallel corpus in the creation of MT models. Soʻz sanʼati xalqaro jurnali, 7(3), 80–85. ISSN: 2181-9297. SCIENCE AND INNOVATION INTERNATIONAL SCIENTIFIC JOURNAL VOLUME 4 ISSUE 12 DECEMBER 2025 ISSN: 2181-3337 | SCIENTISTS.UZ 91