Application of Language Models for the Functional Annotation of Conserved Domains in Biological Data

Laboratorio de Investigación Aplicada, Departamento de Informática y Computación, UTEM; Escuela de Informática, UTEM;. This work was supported in part by Project supported by the “Competition for Research Regular Projects”, year 2023, code LPR23-09 and “Competition for Research Assistant Funding UTEM”, year 2023, code AI23-06, Universidad Tecnológica Metropolitana (AM-B) Modern LLMs effectively recognized functionally important regions in amino acid sequences when enhanced with appropriate techniques. RAG demonstrated that this conserved domain recognition capability is determinant for COG functional classification, achieving an F1 score of 0.258. The improvement obtained using the RAG method revealed that while providing more information enables the model to better recognize functional categories, the model exhibited uncertainty and bias toward prompt examples despite conducting correct sequence analyses. Integration of additional metadata such as "Footprint" or "Professional Analyses" could potentially enable more confident model predictions and reduce bias. DeepSeek R1:1.5B DeepSeek LLM:7B Llama 3.2:1B Llama3.2:3B Qwen 2.5:7B Qwen 3:8B Mistral 7B Y. Ji, Z. Zhou, H. Liu, and R. V. Davuluri, ``DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome,'' Bioinformatics, vol. 37, no. 15, pp. 2112--2120, 2021. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention Is All You Need,'' arXiv preprint arXiv:1706.03762, 2017. J. Meier, R. Rao, R. Verkuil, J. Liu, T. Sercu, and A. Rives, ``Language models enable zero-shot prediction of the effects of mutations on protein function,'' bioRxiv preprint bioRxiv:2022.07.20.500902, 2022. Y. Gao and others, ``Retrieval-Augmented Generation for Large Language Models: A Survey,''arXiv preprint arXiv:2312.10997, 2024. E. J. Hu and others, ``LoRA: Low-Rank Adaptation of Large Language Models,''arXiv preprint arXiv:2106.09685, 2021. T. Dettmers and others, ``QLoRA: Efficient Finetuning of Quantized LLMs,''arXiv preprint arXiv:2305.14314, 2023. Application of Language Models for the Functional Annotation of Conserved Domains in Biological Data Hugo Osses Prado ([email protected]), Raul Caulier Cisternas ([email protected]) and Ana Moya-Beltrán ([email protected]) 1 2 2 Escuela de Informática, Facultad de Ingeniería, Universidad Tecnológica Metropolitana, Santiago, Chile. 1 Departamento de Informática y Computación, Facultad de Ingeniería, Universidad Tecnológica Metropolitana, Santiago, Chile. 2 Introduction: Confusion Matrix: Performance Metrics: Conclusions: Acknowledgments: Methodology: This study employs a systematic three-phase approach to evaluate LLM strategies for functional genomic annotation, as illustrated in the experimental workflow design (Figure 2). Phase 1 involves downloading the COG database, conducting exploratory data analysis, and preparing standardized datasets for machine learning applications. Phase 2 implements three distinct approaches: fine-tuning creates specialized COG models using QLoRA/LoRA techniques, prompt engineering evaluates zero-shot capabilities with balanced datasets, and the RAG system combines embedding processes with a comprehensive database for dynamic information retrieval. Phase 3 conducts systematic performance evaluation across all approaches using precision, recall, F1-score, and efficiency metrics, providing comparative analysis and evidence-based recommendations. This framework ensures robust and comparable results by maintaining consistent protocols throughout the experimental pipeline, from data preparation through model implementation to comprehensive performance assessment. Transformers Arquitecture: The transformer architecture serves as the foundation for modern LLMs, as demonstrated in the architectural diagram (Figure 1), utilizing attention mechanisms to determine data importance through key components including encoders for input processing, decoders for output generation, multi-head attention for capturing relevant information, feed-forward networks for data transformation, normalization layers for training stability, and positional encoding for token position information. Different LLM families implement distinct architectural innovations and training approaches: DeepSeek employs iterative reasoning architectures for complex problemsolving, LLaMA maintains balanced designs for general-purpose applications, Qwen features expanded transformer architectures for enhanced capacity, and Mistral specializes in advanced attention mechanisms. Models Used: The confusion matrices reveal distinctive bias patterns between strategies. Zero-Shot (Figure 5) and One-Shot (Figure 4) show bias toward specific categories, with One-Shot more pronounced by "sticking" to given examples. FineTuning (Figure 6) exhibits concentrated patterns with marked bias due to overfitting. RAG (Figure 3) demonstrates better prediction distribution across categories, explaining its superior F1 score of 0.257 and greater generalization capacity, validating its effectiveness in reducing categorical bias for annotation. Experimental evaluation revealed distinct performance patterns across four strategies. Zero-Shot and One-Shot Metrics (Figures 7 and 9) showed Deepseek-R1 1.5b achieving highest accuracy (6%) while Llama3.2 models led F1 scores, with One-Shot improving over Zero-Shot. FT Metrics (Figure 8) demonstrated modest improvements with QLoRA significantly enhanced computational efficiency (0.079). RAG Metrics (Figure 10) achieved the breakthrough with 24.5% accuracy using Llama3.2 3b, substantially outperforming all other strategies and validating the effectiveness of combining RAG with ESM-2 embeddings. References: Figure 3: RAG Confunsion Matrix Figure 4: One-Shot Confunsion Matrix Figure 5: Zero-Shot Confunsion Matrix Figure 6: Fine-Tuning Confunsion Matrix Figure 8: Fine-Tuning Metrics. Figure 9: One-Shot Metrics. Figure 10: RAG Metrics. Figure 2: Experimental desing of the Workflow Figure 1: Transformers Arquitecture Large Language Models (LLMs) have revolutionized natural language processing through their transformer-based architectures, demonstrating unprecedented capabilities in text generation, translation, and information retrieval. These advanced computational models are increasingly being applied to computational biology, showing significant promise in protein structure prediction, genetic variant classification, and biological sequence annotation. Functional annotation enables researchers to identify how these organisms interact with each other and with host systems, providing crucial insights for disease prevention and treatment strategies. However, the analysis of pathogenic organisms generates massive volumes of biological data that require automated processing to identify functions, roles, and characteristics efficiently. Gene functions, organized through structured databases like NCBI COG, encompass fundamental biological processes essential for understanding life, health, and disease. This research addresses the critical challenge of efficiently annotating biological sequences and predicting protein functions by leveraging the advanced language understanding capabilities of LLMs. By improving the accuracy and efficiency of functional genomic annotation, this study aims to accelerate discoveries in biotechnological and pharmaceutical research, ultimately contributing to better understanding and treatment of priority health conditions identified by global health organizations. Figure 7: Zero-Shot Metrics.