It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29533–29548 November 4-9, 2025 ©2025 Association for Computational Linguistics It’s All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs Yue Li, Zhixue Zhao and Carolina Scarton Department of Computer Science, University of Sheffield, UK {yli381,zhixue.zhao,c.scarton}@sheffield.ac.uk Abstract Extremely low-resource languages, especially those written in rare scripts, as shown in Figure 1, remain largely unsupported by large language models (LLMs). This is due in part to compounding factors such as the lack of training data. This paper delivers the first comprehensive analysis of whether LLMs can acquire such languages purely via in-context learning (ICL), with or without auxiliary alignment signals, and how these methods compare to parameter-efficient fine-tuning (PEFT). We systematically evaluate 20 under-represented languages across three state-of-the-art multilingual LLMs. Our findings highlight the limitation of PEFT when both language and its script are extremely under-represented by the LLM. In contrast, zero-shot ICL with language alignment is impressively effective on extremely low-resource languages, while fewshot ICL or PEFT is more beneficial for languages relatively better represented by LLMs. For LLM practitioners working on extremely low-resource languages, we summarise guidelines grounded by our results on adapting LLMs to low-resource languages, e.g., avoiding fine-tuning a multilingual model on languages of unseen scripts. 1 Introduction Current large language models (LLMs) are typically pre-trained with data in more than 50 languages, offering robust support for high-resource languages, such as German and French (Le Scao et al.,2023;Grattafiori et al.,2024;Team et al., 2023). However, their coverage of low-resource languages remains limited. Since these languages are often spoken in developing regions, insufficient LLM support risks reinforcing socio-economic disparities and further isolating affected communities (Shen et al.,2024;Jadhav et al.,2024). 1https://cloud.google.com/translate/docs/ languages Figure 1: Regional distribution of the languages studied in this paper. Red denotes the five languages with rare scripts, and blue represents the other 15 languages. Y(yes) and N(no) denote whether it’s supported by Google Translate 1 .Accuracy represents performance on topic classification (SIB-200) with DeepSeek (7b) in zero-shot ICL (majority voting = 0.25, English = 0.83). Extending LLMs to support extremely lowresource languages via continued pre-training (Yong et al.,2023a) is possible but often impractical, due to the need for large-scale monolingual corpora and substantial computational resources (Joshi et al.,2020). Although LLM support can be achieved for downstream tasks via resourceefficient training, such as parameter-efficient finetuning (PEFT), it still requires a non-trivial amount of labeled data. Therefore, with recent advances in in-context learning (ICL), we ask whether LLMs can learn new languages purely through ICL (Yong et al.,2023b;Zhang et al.,2024;Cahyawijaya et al.,2024). Specifically: (1) Is ICL alone sufficient to enable LLMs learn extremely lowresource or entirely unseen languages? (2) Can auxiliary signals in the prompt be useful enabling or improving ICL? (3) ICL or PEFT, which one is 29533 a better choice for learning a new language? In this study, we consider 20 low-resource languages, including five extremely low-resource ones (Figure 1) and 15 written in Latin, Arabic, or Cyrillic scripts (referred to as target languages 2 ). ICL with auxiliary signals (i.e., class category, language alignment) is explored. Our setup (Table 1) includes few-shot ICL, sentence-level alignment of unlabelled or labelled examples in zero-shot or fewshot ICL, and word-level alignment for the target language in zero-shot ICL. To our knowledge, this is the first study to systematically analyse whether ICL can enable LLMs to learn extremely low-resource languages 3 . Our main findings are: • In contrast to prior work (Razumovskaia et al., 2024), small-scaled fine-tuning is generally ineffective when a language and its script are highly under-represented or entirely absent from both the tokenizer and pre-training data (e.g., sat , nqo and taq on DeepSeek). • In such cases, zero-shot ICL with language alignment yields substantial gains, potentially surpassing vocabulary extension on multilingual pretrained language models (PLMs) through continue pre-training. • Zero-shot ICL with language alignment is especially effective for languages with minimal LLM support, often exceeding few-shot ICL and performing comparably to, or better than, finetuning. • Few-shot ICL and especially PEFT perform best for low-resource languages for which LLMs exhibit a certain level of support. 2 Related Work Language Adaptation in Pretraining Continued pretraining LLMs on a monolingual corpus in a target language is a common strategy to extend support to languages not (well) represented in the original pretraining data, also enhancing ICL performance in the target language (Yong et al., 2023a). Various methods have been explored for training efficiency (Zhang et al.,2021;Yong et al., 2023a;Cui et al.,2023), vocabulary and tokeniser adaptation (Yamaguchi et al.,2024a;Balachandran, 2represented by language code ISO 639-3 (in Table 2) 3 The term "extremely low-resource languages" refers to the five languages with rare scripts in this study 2023;Cui et al.,2023;Larcher et al.,2023) and data efficiency (Yamaguchi et al.,2024b;Shaham et al., 2024;Kurz et al.,2024). However, the effectiveness of these pretraining-based methods often depends on the availability of large-scale training data, an assumption that does not hold for extremely lowresource languages in real-world scenarios. Adapting LLMs to Low-Resource Languages for Downstream Tasks ICL with different strategies has been explored to improve LLMs’ adaptation to low-resource languages, including techniques such as code-switching (Yong et al.,2023b; Schlicht et al.,2025), demonstration example selection (Winata et al.,2022;Zhang et al.,2024; Tanwar et al.,2023), prompt format optimization (Zhang et al.,2023;Cahyawijaya et al.,2024), machine translation (Bandarkar et al.,2024), and dictionary-based prompting (Lu et al.,2024;Zhang et al.,2024). Another promising direction is PEFT, which has demonstrated superior performance with computational costs comparable to few-shot ICL (Liu et al.,2022). However, most existing studies focus on languages that are: (1) relatively highresource (e.g., German); (2) low-resource but written in widely supported scripts (e.g., Zhuang in Latin script (Zhang et al.,2024)); and (3) written in rare scripts but already included in model pre-training (Razumovskaia et al.,2024). Consequently, how to effectively adapt LLMs to extremely low-resource languages such as nqo (Figure 1) is still unclear. 3 Learning Extremely Low-Resource Languages Table 1summarises the experimented approaches and assumed available data resources. Standard cross-lingual transfer that aims to improve and then transfer task knowledge from English is not considered in this study (i.e, fine-tuning with English data or few-shot ICL with English examples), as the LLMs have already demonstrated high accuracy on English in zero-shot ICL (Table 4)4. Baseline The vanilla zero-shot ICL when the LLMs are prompted only with task description and the target language input tgt . We use English task 4 Machine translating target-languages into English is not considered, since our aim is to teach low-resource languages to LLMs. Reliable machine translators are also not available for 3 out of 5 rare-script languages. In practice, developing a highquality machine translator is significantly more expensive than creating the data resources we consider here. 29534 Method Data Training Annotator EN Target Other Zero-shot ICL baseline ✗ ✗ ✗ ✗ ✗ sentence-level alignment ✓(k)✓(k)✗ ✗ translate word-level alignment ✗ ✗ dict. ✗ ✗ word-level translation ✗ ✗ dict. ✗ ✗ Few-shot ICL demonstration in target language ✗✓(k)✗ ✗ label demonstration with alignment ✓(k)✓(k)✗ ✗ label+translate Parameter efficient fine-tuning ✗✓(N)✗✓label Table 1: List of methods used in this paper and resources they may rely on: (1) Data: in-domain data in English (EN) or target language (Target), or a dictionary (dict.) for word translation; k denotes k -shot examples, and N denotes the full training data ( k≪N ); (2) Training: whether model parameters are updated; (3) Annotator: whether native speakers are needed to label the data, or translate data to English to enable alignment. description as it has been widely adopted showing improvements in ICL performance (Zhang et al., 2023;Razumovskaia et al.,2024). The prompt format is: [<task description> + <inputtgt>]. Zero-Shot ICL with alignment we experiment with adding wordor sentence-level alignment between English and a target language in the prompt, without providing labelled examples. • Word-level alignment: We provide a translation for each word in the target-language input using a dictionary, inspired by prior work on machine translation (Zhang et al.,2024;Lu et al.,2024). The prompt format for an input tgt with N words {w tgt 1 , w tgt 2 , ..., w tgt N } is: [<task description> + <input tgt > + <w tgt 1 means w eng 1 in English; ...; w tgt N means w eng N in English>]. We use the NLLB translator 5 (Costa-Jussà et al.,2022) to create the dictionaries following Lu et al. (2024). For languages not supported by NLLB ( nqo , sat6 , and min ), we train the word alignment tool fast_align (Dyer et al.,2013) to simulate a high-quality dictionary (See Appendix B). • Word-level translation: We directly concatenate the English word translations in their orders in target languages as the “English” translation (i.e., input eng′ = concat (w eng 1 , ..., w eng N )), and prompt LLMs with: [<task description> + inputeng′]. • Sentence-level alignment: Assuming there is a limited k number of parallel in-domain unlabelled sentences in English {s eng 1 ,...,s eng k } and target language {s tgt 1 ,...,s tgt k }. The prompt format is: [<target language: s tgt 1 ; English: s eng 1 ; ...; target language: s tgt k ; English: s eng k > + <task 5https://huggingface.co/facebook/nllb-200-3. 3B 6NLLB repeats the same word without stopping description> + <input tgt >]. We select the targetlanguage example sentences from the training data through random sampling or BM25 (Robertson et al.,2009). Few-Shot ICL Assuming there is a limited number of labelled in-domain data samples in the target language or English, demonstration examples from the training data are retrieved by BM25, inspired by Zhang et al. (2024). • Demonstration in target language: We prompt LLMs with k -shot demonstration examples in the target language D tgt 1 , ..., D tgt k . The prompt format is [<task description> + <D tgt 1 , ..., D tgt k > + <inputtgt>]. • Demonstration with alignment: LLMs are prompted with parallel demonstration examples in both English and target languages: [<task description> + <D tgt 1 means D eng 1 , ..., D tgt k means Deng k> + <inputtgt>]. PEFT We preliminarily experiment with competitive methods such as LoRA (Hu et al.,2022), DoRA (Liu et al.,2024) and IA3 (Liu et al.,2022). Same as Yong et al. (2023a), we found that IA3 is the most effective and efficient approach. Therefore, due to computational constraints, we only experiment with IA3 as a representative of the PEFT methods. We also discuss the comparison with fully fine-tuned multilingual PLMs in Section 4.4. 3.1 Experimental Setups Target Languages We mainly ground our research on the SIB-200 seven-way topic classification dataset (Adelani et al.,2024), as it offers parallel training and evaluation data with the broadest multilingual coverage in natural language understanding (NLU) tasks. We also analyse the generalisability of our findings on reading comprehension 29535 (i.e., BELEBELE (Bandarkar et al.,2024)), which is a more challenging task than topic classification. Since most prevalent LLMs do not disclose comprehensive lists of languages present in their pretraining data, we select the low-resource languages for which LLMs exhibit significantly limited capability. We measure the LLMs’ capability on each language with Information Parity (IP) (Tsvetkov and Kipnis,2024) on the SIB training data. Given a text in the target language and its English translation, IP is defined as the ratio between the negative log-likelihood of the target-language text and that of its English counterpart under the same model. A very low IP score indicates that the LLM struggles to represent information in the target language, likely due to limited or no exposure during pretraining. Based on this criterion, we select the languages with the average lowest IP scores across the LLMs we study on. This includes five languages written in relatively rare and distinct scripts (Figure 1) plus 15 languages using more commonly supported scripts (i.e., seven in Latin, four in Arabic, and four in Cyrillic). For the latter, we select languages that do not have the same linguistic roots as English (Latin script), Modern Standard Arabic (Arabic script) and Russian (Cyrillic script), which are commonly represented in LLMs’ training data. The full list of the languages is shown in Table 2. Models We experiment with three recent opensource instruction-tuned LLMs with multilingual ability: DeepSeek 7 ,LlaMA-3.2 8 and Gemma-2 9 . Due to computational constraints, their mediumsized variants are considered. Setups We adopt accuracy as the evaluation metric following Adelani et al. (2024). We use greedy search in decoding for the purpose of reproducibility. The prompt template for baseline zero-shot ICL is also used in PEFT for a fair comparison. SIB200 dataset’s official train/dev/test set split is used. The examples included in zero-shot and few-shot ICL are retrieved from the training data. As preprocessing for BM25, only white-space splitting is applied. The hyper-parameter tuning and training details for IA3 are included in Appendix B. 7https://huggingface.co/deepseek-ai/ deepseek-llm-7b-chat 8https://huggingface.co/meta-llama/Llama-3. 2-3B-Instruct 9https://huggingface.co/google/gemma-2-2b-it 4 Results 4.1 Limitation of Fine-Tuning Fine-tuning Improvement Disparity Figure 2 illustrates the performance improvement after finetuning with the training data in the target language. In most cases, fine-tuning leads to enhanced performance, although the degree of improvement varies notably. For low-resource languages using common scripts, accuracy scores can rise to more than 0.6 on average, resulting in an acceptable performance (full results in Table 4). In contrast, results on the five languages written in rare scripts are inconsistent. For instance, while DeepSeek performs worse than majority voting on all of these five languages in the baseline zero-shot ICL setting, PEFT raises the accuracy scores of dzo and tir to above 0.45. In contrast, gains for the remaining three languages are rather modest, particularly for sat , which still stays slightly below majority voting. We observe that this discrepancy is due to overfitting, which appears to occur at a very early stage in the fine-tuning for languages showing limited improvement. Risk of Overfitting and Impact Factors To gain more insights on why certain languages suffer more severe overfitting, we analyse: • Tokenization efficiency: Most LLMs’ tokenisers, including those used by the three models in our study, adopt byte-level Byte Pair Encoding (BPE) (Wang et al.,2020) or SentencePiece (Kudo and Richardson,2018). When encountering texts in rare scripts not seen during tokeniser training, characters are often segmented into raw bytes, resulting in a vocabulary with drastically reduced effectiveness. For instance, BPE tokenisers based on UTF-8 encoding may end up representing an entire rare-script language using only 256 raw-byte token values (Wang et al.,2020), limiting the model’s ability to learn generalisable linguistic patterns with small training data (Zhao and Aletras,2024). Tokenisation efficiency for a given text i is measured using Token-to-Byte Ratio (TBR) =numi tokens numi bytes , where numi bytes is the number of bytes required to represent the text with the same encoding system used in the tokeniser, and numi tokens is the number of tokens produced by the LLM’s tokeniser. A TBR score close to 1 indicates that the tokeniser is operating nearly at the raw byte level, signalling extremely 29536 (a) DeepSeek (b) LLaMA-3.2 (c) Gemma-2 Figure 2: Accuracy improvement from baseline zero-shot ICL (denoted as × ) to PEFT (denoted as • ). The performance over languages in Latin (latn), Arabic (arab), and Cyrillic (cyrl) scripts is averaged. The performances of English (eng) and the majority voting baseline (vertical dashed line, accuracy = 0.25) are for reference. (a) DeepSeek (b) LLaMA-3.2 (c) Gemma-2 Figure 3: Correlation between information parity (IP), token-to-byte ratio (TBR) and accuracy score after fine-tuning. For improved visualisation, the x-axis represents ( 1− TBR). The language bubbles close to the left corner denote languages that are under-represented by both tokeniser and model. Bubbles with darker colour denote lower performance after fine-tuning. Names of the languages with PEFT performances lower than 0.4 are marked in red. poor tokenisation. For example, the average TBR for sat ’s training data with DeepSeek’s tokeniser is 0.99, suggesting that nearly every character is segmented into raw bytes and that DeepSeek significantly lacks meaningful representations for sat. • Multilingual capability: Fine-tuning is usually more effective when the LLM has already acquired some linguistic competence in the target language during pre-training. IP is used again to estimate the LLMs’ capabilities for each target language prior to fine-tuning. As discussed in Section 3.1, the higher the IP value the more efficiently the LLM represents information provided in the target language. Conversely, a low IP score suggests under-representation of a language. Figure 3shows the average TBR and IP scores for each target language in the training data, also presenting their correlations with fine-tuning performance. Languages with high TBR and low IP scores are the ones where fine-tuning tends to encounter more severely overfitting, resulting in very limited generalisation. This suggests that smallscaled fine-tuning on downstream tasks is unlikely to be beneficial when a language and its script are highly under-represented or even unseen in both tokeniser training and model pre-training. This finding also highlights the importance of improving representation of low-resource languages and scripts during pre-training. Even modest improvements in representation, either at tokeniser or model pretraining level, can lead to notably more effective find-tuning adaptation. For instance, although tir , sat , and nqo are nearly entirely tokenised as raw bytes by DeepSeek’s tokeniser (TBR > 0.99), DeepSeek exhibits stronger pre-trained capabilities on tir (higher IP) compared to sat and nqo , translating into more substantial gains from fine-tuning. Similarly, while LLaMA-3.2 shows comparable IP scores for both dzo and tir , dzo benefits from better tokenisation (lower TBR, i.e. higher 1 − TBR), potentially leading to larger performance improvement after fine-tuning. 4.2 Alignment in Zero-Shot ICL 4.2.1 Sentence-Level Alignment Effectiveness of Semantic Similar Examples Figure 4shows DeepSeek’s performance when prompted with one unlabeled example in the target 29537 (a) Rare Scripts (b) Latin (c) Arabic (d) Cyrillic Figure 4: Accuracy scores for DeepSeek in zero-shot ICL with sentence-level alignment when one unlabelled sentence is BM25-based (green) or randomly sampled (blue). Red denotes baseline zero-shot ICL. language alongside its English translation. Similar trends are observed for LLaMA-3.2 and Gemma-2, with detailed results in the Appendix C. Although the topic label of the example is not included in the prompt, incorporating semantically similar texts in both target language and English significantly enhances performance for low-resource languages with low baseline zero-shot ICL performances, especially those using rarer scripts. However, this benefit diminishes as the baseline zero-shot ICL performance improves. For languages with baseline accuracy scores below 0.3, all three LLMs show an average improvement exceeding 0.22, with a peak gain of 0.36 for sat on LLaMA-3.2. In contrast, for languages with baseline scores above 0.3, the average improvement falls below 0.07. In some cases, LLaMA-3.2 and Gemma-2 exhibit minor performance declines, with the largest drops being 0.1 for kaz on both LLaMA-3.2 and Gemma-2 (baseline = 0.59 and 0.61 respectively). Furthermore, the models’ performance often degrade when examples are randomly sampled from the training set, highlighting that the effectiveness of the alignment hinges on the semantic similarity between the input text and the example in target language. Moreover, improving semantic search for low-resource languages or even enhancing text pre-processing approaches (e.g., lemmatisation and stemming), beyond the simple whitespace-based tokenisation used here, has the potential to increase performance in this sentence-alignment setting. Impact of the Number of Examples We further analyse the performance when varying the number of unlabelled parallel examples provided in the prompt, from two to five examples. We find that increasing the number of randomly sampled parallel examples in the prompt could slightly improve the performance, although the gap between BM25 sampled examples is still notable. However, providing more semantic related examples does not consistently improve performance across all languages. We hypothesize that it may be influenced by the relative length of the target-language sequences compared to English. To test this, we compute the average tokenizer parity (TP) scores (Petrov et al., 2023) for each language in the training set (in Table 2). Given a text in its target language and English translation, TP is defined as the ratio of the number of tokens in English to the number of tokens in the target language. A lower TP score indicates that the target language is tokenized into relatively longer sequences compared to English. We define a binary variable indicating whether adding more examples is beneficial: it is set to true if at least 3 out of the 4 multi-shot settings (2, 3, 4, and 5 examples) outperform the 1-shot setting. We then calculate the point-biserial correlation coefficient (Lev,1949) between the TP score and this binary indicator. The results show a statistically significant correlation, suggesting that languages with lower TP scores are less likely to benefit from additional parallel unlabelled examples. 4.2.2 Word-Level Alignment (a) chrf++ ≤0.5 (b) chrf++ >0.5 Figure 5: Accuracy scores for LLaMA-3.2 over lowresource languages in zero-shot ICL with word-level alignment (gray) or word-level translation (orange) settings. Red denotes baseline zero-shot ICL. The zero-shot ICL performance with word-level alignment or translation on LLaMA-3.2 is shown in Figure 5. The reported chrf++ score 10 of each language on NLLB is used as an indicator of the qual10https://tinyurl.com/nllb200dense3bmetrics 29538 ity of the dictionary. For DeepSeek and Gemma-2 (full results in Appendix C), adopting word-level alignment or translation always improve the performance over the baseline. However, LLaMA3.2’s performance is highly influenced by the dictionary’s quality. When the chrf++ score is lower than 0.5 (Figure 5a), including the low-quality word-level alignment (gray lines) can harm the performance (e.g., fuv , kac , wol , and azb ). In contrast, when chrf++ score is higher than 0.5 (Figure 5b), word-level alignment is always beneficial. For nqo , sat , and min , we train fast_align on the SIB-200 dataset to simulate a high-quality dictionary (see Section 3). LLMs’ performance for these languages is always improved with wordlevel alignment (around 0.6 of accuracy improvement), highlighting the importance of the dictionary quality. We provide discussion on potential impact of fast_align quality in the Appendix C. ICL with word-level translation significantly reduces the inference time than using word-level alignment by reducing input length. However, its performance exhibits variable superiority across languages and LLMs. Specifically, it is consistently better than world-level alignment on LLaMA-3.2, while it is always worse than world-level or even baseline on DeepSeek and Gemma-2. 4.3 Alignment in Few-Shot ICL For languages with baseline accuracy lower than majority voting, including English translations in few-shot ICL often improves results across all the LLMs, when using more than one demonstration example. However, in 1-shot ICL, removing the English translation tends to yield better performance on DeepSeek and LLaMA-3.2, while Gemma-2 continues to benefit from them. For languages with strong zero-shot ICL performance, both DeepSeek and Gemma-2 benefit from the alignment, whereas LLaMA-3.2 performs best without them. Overall, unlike the consistent trends across LLMs in zero-shot ICL with alignment, we observe more variations how LLMs respond to aligned prompts in few-shot scenarios (full results in Appendix C). Although not examined by prior work (Cahyawijaya et al.,2024), we find that model performance can be highly sensitive to the order of the task description and demonstration examples in the prompt, for certain languages, especially on DeepSeek. For example, when prompting Deepseek with 1-shot labelled nqo example with English translation, the accuracy jumps from 0.30 (a) extremely low (b) low (acc<0.45) (c) low (acc>0.45) Figure 6: Accuracy comparison among baseline (red), PEFT (green), best zero-shot ICL (blue) and best fewshot ICL (black) on LLaMA-3.2. Languages are categorised into: (a) Both language and script are severely under-represented (names in red in Figure 3b, baseline accuracy < 0.2): zero-shot > few-shot > PEFT; (b) Better represented, but baseline < 0.45: PEFT > zero-shot ≥ few-shot; (c) Baseline > 0.45: PEFT > few-shot >zero-shot. to 0.42 if the task description is moved to after the demonstration example. In most of the cases, prompting with demonstrations at the beginning lead to better performance for DeepSeek and Gemma-2, while LLaMA-3.2 slightly prefers task description at the beginning. Results presented for few-shot ICL are based on the optimal task description position selected on the validation set. 4.4 Comparison across PEFT, Zeroand Few-Shot ICL Based on our analysis, zero-shot ICL with word11 or sentence-level alignment and few-shot ICL with or without alignment can all achieve promising improvement over the baseline across languages and LLMs. Next, we discuss which approach is more effective from different aspects. We present the results in Figure 6for LLaMA-3.2. DeepSeek and Gemma-2 show same trend (see Appendix C). Fine-Tuning vs. ICL When the low-resource language is extremely under-represented in both tokeniser and model (Figure 6a), fine-tuning LLMs or even PLMs 12 yields minimal improvement, while zero-shot ICL with either sentenceor word-level alignment offers significant improvements (Figure 2a vs. 4a). Adelani et al. (2024) extended the vocabulary of XLM-R for nqo with continuepretraining, leading to fine-tuning accuracy on SIB200 test set rising 0.17 points. However, it is still lower than the performances of LLMs in zero-shot ICL with alignment, which are above 0.41. 11 Results based on fast_align are excluded, as it potentially leaks gold standard English translations into the prompt. 12 Based on results for XLM-R (large) from Adelani et al. (2024), see Table 4. 29539 For the remaining low-resource languages (Figure 6b and 6c), fine-tuning normally has better performance than ICL. Overall, the average difference between PEFT and the best ICL approach on DeepSeek, LLaMA-3.2 and Gemma-2 is 0.13, 0.13, and 0.08, respectively. Zero-Shot vs. Few-Shot We observe that for languages with low baseline zero-shot ICL performance (i.e., accuracy < 0.45 on all LLMs), in most cases (at least more than 50%), zero-shot ICL with word/sentence-level alignment leads to better performance than few-shot ICL regardless if the alignment is provided or not (Figure 6b, comparing black and blue lines). When baseline performance is higher than 0.45 (Figure 6c), few-shot ICL always provides best results. In our study, these languages are khk , tgk , azb , eus , tat , kaz , and urd , consistent across all the LLMs. Overall, if the LLM significantly lacks capability on the target language, providing label in ICL might be useless. 5 Discussion 5.1 NLP Tasks beyond Topic Classification Reading Comprehension We test our findings on BELEBELE, a reading comprehension parallel multilingual dataset, covering 11 out of the 20 languages studied here. It contains questions with four multiple-choice answers linked to a passage. tir is available among the five languages with rare scripts. We split the data into training, validation and test, following SIB-200, to enable a consistent comparison (See Appendix C). We conduct experiments only on LLaMA-3.2, due to its long context length and computational constraints. We retrieve the passage from the training data with BM25 as example and provide its English translation as passage alignment. We adopt accuracy for evaluation (Bandarkar et al.,2024). Most results align with our observations on SIB200 dataset. Specifically, PEFT still shows no improvement for tir , while being more effective for other languages. Zero-shot ICL with passage alignment could still improve over the baseline zero-shot ICL in most cases, especially for the languages with lower baseline performance (e.g., accuracy < 0.35). However, as the task is more challenging, the level of improvement is not as notable as on topic classification. The model also potentially requires more unlabelled parallel data for consistent improvements across all languages. Similar to topic classification, in most cases, zero-shot ICL with passage alignment surpasses few-shot ICL when baseline performance is lower than 0.5. Conversely, word-level alignment is not effective on reading comprehension, which may be explained by the quality of the dictionary created. Unlike topic classification, whose prediction can be made based on one or two topic-related words, reading comprehension relies less on such cue words, requiring a higher quality of word translations. Machine Translation We further experiment with the FLORES-200 machine translation dataset (Costa-Jussà et al.,2022). Among the ICL settings we explore in this study (see Table 1), only the fewshot ICL (demonstration with alignment) setting is relevant to machine translation, which is equivelant to standard few-shot ICL for translation 13 . Similar with BELEBELE, we evaluate on the official dev-test set with LLaMA-3.2. We retrieve one demonstration example from the official dev set with BM-25, and adopt the chrf++ score for evaluation. We present the full results in Table 5 (see Appendix C). The results still align with our observations on classification tasks. Only one-shot demonstration example yields improvements in translation quality across both translation directions, except for urd where LLaMA-3.2 achieves significantly higher performance than for other languages in zero-shot ICL. Also, the gains are generally larger when translating from English into target languages than in the reverse direction. However, overall machine translation quality remains poor. For example, the chrf++ score increases substantially from 0.4 to nearly 11 when translating English to sat , but such translation quality is still far from adequate for practical human use. 5.2 Suggestions to Practitioners In practice, performance is not the only consideration. Investment in data and computational resources needs to be carefully considered, especially for low-resource languages. Aiming to adapt an LLM to a low-resource language for a downstream task, which approach should be prioritized, and what types of data should be created? 13 Zero-shot ICL with sentence alignment corresponds to few-shot ICL, as demonstration examples of translations are provided in the prompt. As for zero-shot ICL with word alignment, we cannot rely on fast_align to simulate the dictionary for the three languages that are not supported by NLLB as we did for classification, because it would directly provide the gold-standard English translations to the LLMs. 29540 For low-resource languages that are extremely under-represented in both tokeniser and model (e.g., nqo ) fine-tuning is not effective and zeroshot ICL with alignment shows promising improvements. We suggest prioritizing investment in human translation to create a small-scale in-domain parallel data for zero-shot ICL with alignment. For low-resource languages where LLMs demonstrate limited capability few-shot ICL might lead to better performance than zero-shot ICL with alignment. However, in most cases, these gains are modest and may even come at the risk of performance degradation. With fine-tuning LLM/PLM being effective and with acceptable zero-shot ICL performance for these languages, decisions should be made by comparing the financial costs between human translation (for zero-shot ICL) and human annotation (for fine-tuning). For low-resource languages where LLMs demonstrate a certain level of capability fewshot leads to better performance than zero-shot ICL with alignment. Human annotation tends to be required for a notable improvement for these languages. Practitioners should consider the trade-off between the amount of data to annotate (effective fine-tuning may require more data than few-shot ICL) and the computational costs (LLM inference is more expensive than fine-tuning PLMs). Since low-resource and extremely low resource languages are tokenised into long sequences, including extra word and sentence alignment in the prompt would further increase the computational costs. We estimate the computational costs for all the ICL settings in this study in Table 7(see Appendix D). 6 Conclusion This work provides a systematic analysis on whether ICL can enable LLMs to effectively support extremely low-resource languages on downstream tasks. As some of the key findings that contrast to prior work, we reveal the limitation of fine-tuning when languages and their scripts are both highly under-represented. In such cases, zeroshot ICL augmented with wordor sentence-level alignment yields promising results. Meanwhile, few-shot ICL or PEFT tends to perform better for languages relatively better represented during pretraining. Our study highlights the importance of language and script coverage in LLMs, and the strong potential of ICL for language adaptation. Limitations Although we conducted more than 450 experiments, our study did not include other popular LLMs, such as Mistral and Qwen. Due to our computational constraints and consideration of fair comparison with PEFT on same LLM size, we did not experiment with LLMs with large sizes, such as Gemma-2 (9b) 14 or LLaMA-3.3 (70b) 15 . Future work could explore whether a larger LLM could enable even more improvement in ICL with wordor sentence level language alignment. Due to very limited datasets with parallel data available for these extremely low-resource languages, we covered topic classification and reading comprehension in this study. On reading comprehension, we were only able to experiment with 11 of the 20 target languages. Both SIB-200 and BELEBELE are constructed based on Flores-200 dataset (Costa-Jussà et al.,2022). With the increase of language coverage for NLP tasks in the future, our findings could be tested on other tasks (e.g., common-sense reasoning or summarization.) and other domains (e.g., medical, social media). As a lack of native speakers and reliable goldstandard word translations, we were not able to accurately access the quality of the dictionary that we created using NLLB translator or fastalign. Our study does not show promising results when using the created dictionary to assist reading comprehension in ICL. However, as discussed in the main content, the results might be improved with a better dictionary. But the effectiveness on SIB-200 and ineffectiveness on BELEBELE imply that for challenging tasks with long input length, word translation quality is more important than for tasks with shorter input. Also, we directly translate words into English to simulate a dictionary following prior work. However, in practice, using a real-world dictionary raises additional challenges such as handling lexical ambiguity and polysemy, which may also impact the performance of word-level alignment in zero-shot ICL. As very limited parallel corpus available for our target languages, we did not systematically analyse how the ICL performance would be impacted if the included unlabelled parallel text is out-of-domain 14https://huggingface.co/google/gemma-2-9b 15https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct 29541 Model Language Code Zero-Shot Few-Shot baseline word-alignment word-translation sentence-alignment without align with align LLaMA-3.2 taq_Tfng 1.58E+13 1.74E+13 1.62E+12 3.31E+13 3.15E+13 3.31E+13 dzo_Tibt 1.93E+13 2.09E+13 1.62E+12 4.02E+13 3.86E+13 4.02E+13 nqo_Nkoo 1.74E+13 1.90E+13 1.62E+12 3.63E+13 3.47E+13 3.63E+13 sat_Olck 1.99E+13 2.16E+13 1.62E+12 4.15E+13 3.99E+13 4.15E+13 tir_Ethi 1.21E+13 1.38E+13 1.62E+12 2.59E+13 2.43E+13 2.59E+13 nus_Latn 6.19E+12 7.81E+12 1.62E+12 1.40E+13 1.24E+13 1.40E+13 ayr_Latn 3.39E+12 5.01E+12 1.62E+12 8.39E+12 6.77E+12 8.39E+12 kac_Latn 3.61E+12 5.23E+12 1.62E+12 8.84E+12 7.22E+12 8.84E+12 luo_Latn 2.89E+12 4.51E+12 1.62E+12 7.41E+12 5.79E+12 7.41E+12 fuv_Latn 2.91E+12 4.53E+12 1.62E+12 7.43E+12 5.81E+12 7.43E+12 wol_Latn 2.97E+12 4.59E+12 1.62E+12 7.55E+12 5.93E+12 7.55E+12 eus_Latn 2.97E+12 4.59E+12 1.62E+12 7.56E+12 5.94E+12 7.56E+12 ckb_Arab 6.31E+12 7.93E+12 1.62E+12 1.42E+13 1.26E+13 1.42E+13 min_Arab 4.51E+12 6.13E+12 1.62E+12 1.06E+13 9.03E+12 1.06E+13 azb_Arab 3.03E+12 4.65E+12 1.62E+12 7.68E+12 6.06E+12 7.68E+12 urd_Arab 4.87E+12 6.49E+12 1.62E+12 1.14E+13 9.75E+12 1.14E+13 khk_Cyrl 5.08E+12 6.70E+12 1.62E+12 1.18E+13 1.02E+13 1.18E+13 tat_Cyrl 4.89E+12 6.51E+12 1.62E+12 1.14E+13 9.79E+12 1.14E+13 kaz_Cyrl 4.83E+12 6.45E+12 1.62E+12 1.13E+13 9.66E+12 1.13E+13 tgk_Cyrl 4.62E+12 6.24E+12 1.62E+12 1.09E+13 9.24E+12 1.09E+13 Gemma-2 taq_Tfng 8.37E+12 9.99E+12 1.62E+12 1.84E+13 1.67E+13 1.84E+13 dzo_Tibt 6.47E+12 8.09E+12 1.62E+12 1.46E+13 1.29E+13 1.46E+13 nqo_Nkoo 9.62E+12 1.12E+13 1.62E+12 2.09E+13 1.92E+13 2.09E+13 sat_Olck 8.07E+12 9.69E+12 1.62E+12 1.78E+13 1.61E+13 1.78E+13 tir_Ethi 5.30E+12 6.92E+12 1.62E+12 1.22E+13 1.06E+13 1.22E+13 nus_Latn 4.54E+12 6.16E+12 1.62E+12 1.07E+13 9.08E+12 1.07E+13 ayr_Latn 3.12E+12 4.74E+12 1.62E+12 7.87E+12 6.25E+12 7.87E+12 kac_Latn 3.27E+12 4.89E+12 1.62E+12 8.15E+12 6.53E+12 8.15E+12 luo_Latn 2.72E+12 4.34E+12 1.62E+12 7.06E+12 5.44E+12 7.06E+12 fuv_Latn 2.61E+12 4.23E+12 1.62E+12 6.84E+12 5.22E+12 6.84E+12 wol_Latn 2.73E+12 4.35E+12 1.62E+12 7.09E+12 5.47E+12 7.09E+12 eus_Latn 2.68E+12 4.30E+12 1.62E+12 6.99E+12 5.37E+12 6.99E+12 ckb_Arab 4.59E+12 6.21E+12 1.62E+12 1.08E+13 9.19E+12 1.08E+13 min_Arab 3.79E+12 5.41E+12 1.62E+12 9.21E+12 7.59E+12 9.21E+12 azb_Arab 2.84E+12 4.46E+12 1.62E+12 7.31E+12 5.69E+12 7.31E+12 urd_Arab 3.07E+12 4.69E+12 1.62E+12 7.77E+12 6.15E+12 7.77E+12 khk_Cyrl 4.17E+12 5.79E+12 1.62E+12 9.95E+12 8.33E+12 9.95E+12 tat_Cyrl 3.51E+12 5.13E+12 1.62E+12 8.65E+12 7.03E+12 8.65E+12 kaz_Cyrl 3.66E+12 5.28E+12 1.62E+12 8.95E+12 7.33E+12 8.95E+12 tgk_Cyrl 3.92E+12 5.54E+12 1.62E+12 9.47E+12 7.85E+12 9.47E+12 DeepSeek taq_Tfng 1.40E+13 1.57E+13 1.68E+12 2.97E+13 2.80E+13 2.97E+13 dzo_Tibt 2.18E+13 2.34E+13 1.68E+12 4.52E+13 4.35E+13 4.52E+13 nqo_Nkoo 1.74E+13 1.90E+13 1.68E+12 3.64E+13 3.47E+13 3.64E+13 sat_Olck 2.12E+13 2.29E+13 1.68E+12 4.41E+13 4.25E+13 4.41E+13 tir_Ethi 1.33E+13 1.50E+13 1.68E+12 2.83E+13 2.66E+13 2.83E+13 nus_Latn 5.68E+12 7.36E+12 1.68E+12 1.30E+13 1.14E+13 1.30E+13 ayr_Latn 3.52E+12 5.20E+12 1.68E+12 8.73E+12 7.05E+12 8.73E+12 kac_Latn 3.78E+12 5.46E+12 1.68E+12 9.24E+12 7.56E+12 9.24E+12 luo_Latn 3.07E+12 4.75E+12 1.68E+12 7.83E+12 6.15E+12 7.83E+12 fuv_Latn 3.03E+12 4.71E+12 1.68E+12 7.74E+12 6.06E+12 7.74E+12 wol_Latn 3.18E+12 4.86E+12 1.68E+12 8.04E+12 6.36E+12 8.04E+12 eus_Latn 3.14E+12 4.82E+12 1.68E+12 7.96E+12 6.28E+12 7.96E+12 ckb_Arab 8.26E+12 9.94E+12 1.68E+12 1.82E+13 1.65E+13 1.82E+13 min_Arab 6.93E+12 8.61E+12 1.68E+12 1.55E+13 1.39E+13 1.55E+13 azb_Arab 6.02E+12 7.70E+12 1.68E+12 1.37E+13 1.20E+13 1.37E+13 urd_Arab 7.65E+12 9.33E+12 1.68E+12 1.70E+13 1.53E+13 1.70E+13 khk_Cyrl 5.20E+12 6.88E+12 1.68E+12 1.21E+13 1.04E+13 1.21E+13 tat_Cyrl 5.06E+12 6.74E+12 1.68E+12 1.18E+13 1.01E+13 1.18E+13 kaz_Cyrl 5.33E+12 7.01E+12 1.68E+12 1.23E+13 1.07E+13 1.23E+13 tgk_Cyrl 4.75E+12 6.43E+12 1.68E+12 1.12E+13 9.50E+12 1.12E+13 eng_Latn 1.65E+12 3.33E+12 1.68E+12 4.99E+12 3.31E+12 4.99E+12 Table 7: Total FLOPs of each ICL approach for each language on DeepSeek, LLaMA-3.2 and Gemma-2 when inference on SIB-200. 29548