Small Language Models for libraries and computational humanities

Small Language Models for libraries and computational humanities Small Language Models for libraries and computational humanities Practical opportunities and use cases for academic libraries and computational humanities researchers Author: Kaspar Beelen, August 2025 Small Language Models for libraries and computational humanities  2 Foreword The media’s emphasis on generative AI (GenAI) in the last few years has overshadowed the fact that small language models (SLMs) are central to computational research in the humanities and – to a degree – the social sciences. The relationship between technological innovation and knowledge production has a long, complex and often contested history in humanities research. And, with the advent of GenAI, discussions about using technology for research have only intensified. Awareness of GenAI as a research tool is increasing but there is little consensus on how to apply it, and debates often focus narrowly on specific models. At Jisc, we have been interested to see how AI might impact humanities research, having had a keen interest in digital humanities (DH) and knowing full well that DH researchers have been deploying forms of AI for decades, especially natural language processing (NLP). We thought it was timely to focus in on these tools. We want to give librarians, researchers and those in universities who make decisions about AI new insights into the value and importance of SLMs – being run locally, and both more predictable in outputs and more environmentally friendly than their larger cousins. Long before ChatGPT burst onto the scene we were documenting the shift in AI in blog posts, webinars and podcasts focused on what researchers and librarians can do to support those needing to use language models (LMs). We commissioned Kaspar Beelen in the School of Advanced Study, University of London to bring all these strands of enquiry together at a time when many in universities want to support the use of LMs but aren’t sure which ones to select, how best to deploy them or how to advise on their use. Small Language Models for libraries and computational humanities  3 Executive summary This report signposts effective work with language models, especially small models, in collections-based humanities research. We offer a definition of SLMs focused on utility rather than parameters. We offer guidance on their appropriate and ethical use in terms of the data consumed, the reproducibility of outputs, the right models for particular problems, and use of data. The key findings section suggests practical activities to help you make good decisions about using AI in university research. We also look at the implications of these technologies for librarians who support humanities (and potentially social science) research. The literature review focuses on major developments and uses of small models by researchers so librarians can be better informed about these technologies. We offer a technical overview, insights into using SLMs in metadata research and for optical character recognition (OCR) correction. Our report proposes that “to support the production of domain-specific models, libraries can publicly release their collections as data. When this is not feasible, they can allow collection-level access within secure environments”. This concept draws on the collections as data initiative (Padilla et al, 2023) and our webinars on that topic. While large language models (LLMs) are powerful and merit attention, we want to highlight how SLMs remain critical tools for computational humanities researchers and librarians, and are often better aligned with their needs. We address the principal question: "What are the opportunities and use cases for SLMs in humanities research?". We do so from a functional perspective, evaluating language models – and AI more generally – in relation to the tasks they perform. What do they achieve? In essence it is all about asking the right questions at the right time. (Note that it may be helpful to read appendix A, which outlines a number of technical matters, and to refer to the short glossary as you read this report.) Small Language Models for libraries and computational humanities  4 Introduction What distinguishes SLMs from LLMs is their openness, customisability and the fact that we can deploy them locally, on consumer hardware. For these reasons SLMs will continue to be popular tools with researchers. (Throughout this report ‘researchers’ means computational humanities researchers unless otherwise specified.) We argue that SLMs perform well in constrained scenarios and for specified tasks, and they offer reliable, costefficient tools that are friendlier to the environment and preserve privacy. We suggest that LLMs expand the humanities toolbox but won’t necessarily replace SLMs. And, crucially, we show that LLMs don’t necessarily outperform SLMs. We also argue that researchers and librarians can avoid over-reliance on (commercial) LLMs if we articulate the task clearly: what outcomes do we expect and what purposes do we want AI to serve? However, we anticipate changes towards more hybrid workflows that combine SLMs and LLMs, using each type of model as appropriate. Then, we offer actionable suggestions for scholars and librarians. They include: • Orchestrating models in hybrid workflows • Generating synthetic data • Implementing retrieval-augmented generation • Promoting a vision of collections as curated (machine learning) data We also explore uses of metadata in research and technologies that introduce structured data into the use of LMs. How do researchers see language models? A survey of around 5,000 researchers (including, but not limited to, humanists) found that, when asked which generative AI tools they were aware of, most had heard of (90%) or used (81%) Open AI’s ChatGPT. However, after ChatGPT familiarity drops off sharply (Wiley, 2024). While acknowledging that AI will be important to researchers, the Wiley report concedes that their use of it is currently fairly limited. But they anticipate higher use in the future: "Researchers know that developing AI skills will be highly important to them personally in the near future." (Wiley, 2024) Wiley’s report finds that "uncertainty is a major barrier to…making the most of AI’s potential". Overall, it observes that researchers (not only computationally focused ones) share a sense of anticipation mixed with uncertainty but harbour a rather narrow understanding of AI, equating this technology to commercial LLMs. Small Language Models for libraries and computational humanities  5 Structure and methodology Audience This report is primarily for library professionals and (computational) humanities scholars interested in AI, to encourage conversations between those who preserve, curate and provide library collections and those who want to explore the data. We are writing for audiences like those described in our digital collections blog post outcomes of task and finish group investigations into the preparation of datasets for AI: “…the ‘digitally obliged’ (those who need to provide services eg libraries, archives and research support teams) in their role of supporting the ‘digitally curious’ (those who want to know but currently may not).” Sources, methods and definitions We’ve based the report on three sources of information: • A targeted literature review • Proceedings of the Computational Humanities Research workshops • Conversations with experts working in libraries or active as researchers (within and outside academia) Computational Humanities Research (CHR) survey methodology The targeted literature review analyses and describes use cases for LMs in computational humanities research focused on the CHR workshops for 2023 and 2024. CHR proceedings provide a relevant, timely dataset to study how language modelling technologies are being used. We downloaded all papers from the 2023 (49 papers) and 2024 (78 papers) proceedings and scanned each one to assess whether language models were used as part of the methodology. First, we reviewed the abstract and keywords: if these didn’t offer clear clues, we searched for terms like ‘language model’ or specific model names such as ‘BERT’ or ‘GPT’. Based on these criteria we retained 24 papers from 2023 and 36 from 2024 for closer analysis. Conversations with experts and guidance Our conversations with experts were semi-structured. We asked questions about the definition of SLMs, the use of LMs in practice, the anticipated applications and common hurdles or other issues. In square brackets below we include experts’ initials to indicate which observations are built on these conversations. We haven’t used direct quotes given the more informal setting of these exchanges. The guidance signposts to further conversations and work. The author is wholly responsible for the report’s content. Small’ means open and customisable Small Language Models for libraries and computational humanities  6 Usually, model size corresponds to the number of parameters. While parameters are important, the amount of training data and the number of epochs also impact a model’s expressive capacities. However, based on the discussions with experts, we chose not to provide an absolute measure of smallness – for example, by choosing a certain number of parameters as a threshold. Instead, we opted for a contextual definition. The levels of control and customisability are what distinguish ‘small’ from ‘large’ models. We therefore define SLMs as models we can run, train or fine-tune locally on consumer-grade hardware (preferably powered by a graphics processing unit [a GPU]). And by definition SLMs are open-weight models, i.e. those we can download, run and customise. We argue below that this has many important implications in terms of scalability, reproducibility and adaptability. Reading the report If you are less familiar with the technical terminology used throughout you will find it helpful to refer to the glossary in appendix B and the explanations in the technical overview in appendix A. Appendix D contains learning material about working with LMs and AI more generally. Acknowledgements We thank the interviewees who have informed the guidance section of the report, and also our external reviewers for their time and commitment. Small Language Models for libraries and computational humanities  7 Key findings There is no ‘one-size-fits-all’ solution Large commercial models have become dominant in the current understanding of generative AI (GenAI), especially for those with little previous experience with machine learning (Wiley, 2024). However, this poses the risk of tailoring the problems to the tool, instead of vice versa [JG]. Or, as AI consultancy founder Matthew Upson says in his blog post how we think about generative AI: “When all you have is a model like ChatGPT, all problems start to look like LLM problems.” The AI landscape is diverse, so widening our horizon is crucial for selecting the tools that align with research aims as well as ethical and environmental values. Small and large models both have benefits and disadvantages. We can’t provide hard and fast rules that guide librarians and researchers to the ‘optimal’ solution, but we can help the process by posing the right questions [GC, IG and JG]: • What is the application? What is the task you need machine learning for? • What is data? Are there privacy or copyright concerns? How large is the collection you work with? • What level of accuracy do you need? What is ‘good enough’? What amount of error is tolerable? • What resources do you have at your disposal? Do you have access to dedicated infrastructure? Do you have the budget for using commercial services? These questions permeate our approaches below. Determine the task or application you need machine learning for A common theme in conversations with experts was uncertainty about the application of AI tools and the specific use cases in the library context: • The problem is less about technical aspects and more about deciding what’s actually useful [DvS] • There aren’t that many library use cases benefitting from AI yet [IG] • Similarly, in the humanities, researchers are still often vague about what they need AI for. Detailing how machine learning fits in the overall argument is difficult, but necessary [MW] SLMs are optimal for specific use cases as an alternative to LLMs If the task is clear and narrow enough SLMs generally provide a competitive alternative to LLMs at a fraction of the cost, with higher speed and lower environmental impact. The analysis of the computational humanities in appendix A shows that research often focuses on extracting and classifying specific information for which smaller-specialised models still offer good performance [SO]. An example is NuExtract. LLMs don’t necessarily show significant improvements for many core tasks. Only a few use cases are discussed in the workshop proceedings that require LLMs or generative solutions. SLMs are not generic problem solvers. If you need applicability across many domains or problems LLMs are the best option. But do libraries and humanities researchers really need artificial general intelligence [DvS] to solve Small Language Models for libraries and computational humanities  8 specific problems? The sudden popularity of LLMs has inspired fear among librarians and researchers that they may miss out or fall behind [LT], but we argue that doing things differently, by for example relying on SLMs, often offers more appropriate solutions to problems they actually encounter. SLMs excel in classification; experiment to get strong results For classifying information, task-specific SLMs remain competitive and often provide faster and cheaper solutions [DvS and SO]. The main hurdle is the upfront cost: the expertise and time required to produce efficient, specialised SLMs. Unfortunately, picking the right tools requires experimentation (and sometimes failure). It is very hard to know in advance if an SLM will perform well enough – i.e. achieve the expected level of accuracy. Tasks that require a high-level semantic understanding of the text and without strong lexical markers remain hard for SLMs. For example, argument detection and frame classification require a solid grasp of nuance and implicit meanings embedded in the document, something that SLMs generally still struggle with [SO]. SLMs can be used for constrained text generation LLMs excel at generalising and solving problems via text generation. Their strong in-context learning abilities allow them to perform myriad (unseen) tasks. SLMs don’t generalise that well but can be optimised for constrained generative settings such as summarisation [DvS], and by using retrieval augmented generation (RAG) (see below). Examples of the latter include optical character recognition (OCR) and OCR correction, specialised tasks that SLMs can do well [SO]. Deploy retrieval augmented generation by connecting generative AI to specialised databases RAG is a workflow that incorporates LMs to explore databases. It combines information retrieval with text generation: we retrieve information relevant to a query and then use an LM to generate a response based on the query and selected documents. SLMs can be crucial for both steps: • Encoding information – using an LM, each document (or text fragment) is converted to a vector that captures its meaning. Retrieval is about selecting documents whose vector representation is most similar to the (embedded) query • Generating a response – after selecting the most relevant documents a generative model formulates an answer to the query based on the information contained by these texts. We recommend instructing the model to answer ‘I don’t know’ if the retrieved documents are irrelevant to the question RAG allows researchers to interact with databases or other information sources without training a model on the data. Moreover, it applies the strength of (constrained) SLMs – encoding, question-answering and summarisation. SLMs can be especially powerful for summarising information extracted from a database. Compared to keyword search RAG is more versatile and efficient. An interesting example is CORE-GPT, a “question-answering platform that combines GPT-based language models and more than 32 million full-text openaccess scientific articles from CORE” (Pride et al, 2023). Connecting LMs to databases via RAG has demonstrated multiple benefits over using standard tools: it provides links and citations to sources in the database, improves trust and reduces hallucinations. The range of questions users can ask is almost limitless. Small Language Models for libraries and computational humanities  9 Deploy SLM with RAG to save cost and improve trust The CORE-GPT application – as described in the paper – integrates OpenAI’s GPT models for generating answers. This approach proved successful as a prototype, but it would drive up costs significantly when applied in production [PK]. Fortunately, smaller models specifically optimised for RAG have recently begun to demonstrate increasingly reliable performance. In the paper even small reasoners should quote their sources LLM developer Pleias introduces a new generation of small reasoning models optimised using mid-training for RAG. Recently, Pleias released two models – Pleias-RAG-350m and Pleias-RAG-1B – pre-trained on the common corpus (we discuss these in more detail later). These models were then mid-trained on a vast synthetic dataset (for more see appendix A) simulating the retrieval of diverse multilingual texts. Despite their smaller size these models have achieved remarkable results, outperforming their peers and showing that these SLMs are emerging as powerful tools for RAG. Moreover, as the title of Pleias’ paper suggests, RAG models are becoming better at citing sources. Adding provenance information greatly enhances trust in AI systems [PK]. See also Pride et al, 2023. Save cost with optimised SLMs when processing/analysing big data LLMs are great for prototyping or delivering a proof-of-concept [DvS, MW, SO]. They take away many of the (technical) roadblocks and allow you to brainstorm scenarios quickly. Commercial models are likely to provide the best performance with little effort. While larger models circumvent upfront costs they saddle users with ongoing or deferred costs. Therefore, the larger the dataset or application the more worthwhile it is to investigate using optimised SLMs, especially within the constraints of academic and library budgets [PK]. Consider hybrid workflows to avoid over-reliance on LLMs Research and applications are increasingly moving towards a more hybrid set-up that combines models of different sizes and capabilities to achieve a given task [FN]. This strategy is also called ‘model orchestration’, the process of coordinating different machine learning models towards a common application or task. Model orchestration allows us to align model selection based on the complexity of the input: it’s a procedure for automatically routing requests based on the difficulty of the question. Use SLMs to ensure reproducibility An additional benefit of SLMs for research is their reproducibility, which cannot be guaranteed in the case of commercial models. Reproducibility depends on controlling all the variables in the research environment. Commercial models queried via application programming interfaces (APIs) are outside the scientist’s control and tend to change over time [PK]. This problem is compounded by the fact that LLMs are increasingly used for evaluation [PK], and these are outside the grip of the researcher as well. When they are documented and shared properly pipelines built on open models enable researchers to reproduce and improve or correct findings. Releasing code, writing data sheets and producing model cards all enhance transparency and facilitate reproducibility. In this scenario researchers control the model weights and the research environment and so they have more leverage over the model outputs, especially when classifying and embedding information. Small Language Models for libraries and computational humanities  16 The continued dominance of BERT Looking more closely at the types of models, it becomes clear that even in the age of LLMs, BERT (see glossary in appendix B) models dominate the computational humanist’s toolbox (Devlin et al, 2019). Technological change always unfolds in relation to existing practices and skills. These figures show a measured, gradual adaptation to emerging technologies, a process in which new models are explored without discarding existing, effective tools. Researchers often have clearly defined use cases for which smaller models are perfectly adequate. Domain-adaptation of open models Equally important is the widespread community effort to optimise BERT-type models for specific languages, historical periods and research tasks. This process is called domain adaptation and consists of (continued) pretraining of language models on new texts (without labels): • When studying authorship attribution for ancient Greek, (Schmidt et al, 2024) draw on various models finetuned for both ancient and modern Greek • (Bjerring-Hansen et al, 2024) work with language-specific models like DanskBERT to classify 19th century Danish and Norwegian novels into ‘historical’ and ‘contemporary’ categories • Studies focusing on historical semantic change in English rely on models like MacBERTH (Arevalo and Fonteyn, 2021) or those developed by the Living with Machines project (Hosseini et al, 2021) Scholars working with older forms of Dutch can turn to GysBERT (van Dalfsen et al, 2024). Beyond language and period adaptation, BERT models are also commonly fine-tuned for particular tasks – for example, sentiment analysis models feature in studies such as those by Rebora and Vezzani (2023) Adaptation of generative models Decoder-only models are slowly finding their way into computational humanities research. The growing community effort to create languageand domain-specific generative models is gradually permeating the field. For instance, SauerkrautLM is being used to examine the evolution of emotions in German-language poetry, drama and prose between 1850 and 1920 (Konle et al, 2024). SauerkrautLM is derived from Mistral models and trained on a German corpus. Importantly, the model's weights are freely available, allowing the researchers to fine-tune it with their annotated datasets. Other popular models in this category belong to the Llama model family, a series of open models produced by Meta (Bamman et al, 2024; Zhang et al, 2024). Domain adaptation and domain specific knowledge, an opportunity for collaboration Domain adaptation of small language models (SLMs) requires training on large volumes of text data. Given the deep curatorial expertise and domain-specific knowledge found in libraries, developing domain-specific SLMs presents a valuable opportunity for collaboration with humanities scholars [LT]. For instance, historically informed open models serve as essential tools for researchers investigating semantic change or studying specialised forms of language use. In such cases, a strong understanding of the source materials is critical. To support the production of domain-specific models, libraries can publicly release their collections as data. When this is not feasible, they can allow collection-level access within secure environments. Small Language Models for libraries and computational humanities  17 B. Workflows and development – how are these models deployed/adapted? Annotations and transfer learning With transfer learning (see appendix A) researchers tailor pre-trained SLMs to specific tasks. This is especially useful when ‘out-of-the-box’ models – ie those downloaded from the web or accessed via APIs – fail to perform a task or don’t apply to your (historical) domain or language. More often than not, use cases in the (computational) humanities are highly specific with respect to both the domain and the task – for example, the detection of animate machines in Victorian literature (Ardanuy et al, 2020), the aspect-based sentiment in Greek myths (Neugarten et al, 2024) or non-named spatial entities in Swiss German (Kababgi et al, 2024). Types of annotation for model fine-tuning Task adaptation: the process of fine-tuning SLMs on (manual) annotations to improve performance on these specific tasks. Data annotation is still a crucial but often overlooked part of the machine-learning workflow. This step is often the most time-consuming for building machine-learning applications. To learn, models need examples and those examples are mostly produced (or at least curated) by humans. Manual annotation: annotations are usually provided by human labellers. Coll Ardanuy et al (2021) annotated a selection of British newspaper articles with place names (toponyms), which they then used to fine-tune a historical BERT model for entity recognition. Similarly, Ziegler (2024) performed event annotation on early modern administrative records, focusing on identifying ‘roles’ and ‘triggers’ as key elements. Leveraging library metadata: In some papers researchers rely on existing metadata as a proxy for the concepts they want to detect (instead of producing new annotations). For example, Bjerring-Hansen et al (2024) use the MeMo corpus, which contains 859 Danish and Norwegian novels from the late 19th century, leveraging its original metadata classifying each work as historical or contemporary. LLM-based annotation: LLMs can annotate data, which is subsequently used to fine-tune SLMs (van Dalfsen et al, 2024). Annotations for model evaluation Part of the annotated data is usually set aside to evaluate the model’s performance. This is known as the test set. By comparing model predictions to the (human-provided) labels researchers can assess how well the AI replicates their understanding of the concepts under investigation. In some papers annotations are used solely for evaluation. This applies when there are good reasons to believe that an existing out-of-the-box model will perform adequately (Laato et al, 2024). To conclude, transfer learning with annotations remains a useful and popular paradigm for task adaptation within the computational humanities. It allows models to align better with specific questions, collections and tasks. Data enrichment, an opportunity for librarians to work with scholars Annotations and data labelling give librarians and humanities scholars excellent opportunities to design and collaborate on research projects. Encouraging researchers to leverage existing classifications in the records metadata is one strategy. Ryan and Fantoli (2024) rely, for example, on language information to train models that detect translations and multilingual publications. More generally, the curatorial and collection-level knowledge that librarians possess – even when it is not captured by metadata – is an important area of future data enrichment and collaboration with scholars [LT]. Another option is to engage the wider public and enrich heritage data via crowdsourcing. The work of Ridge et al (2022) in the context of the Living with Machines project is an excellent example. Based on annotations provided Small Language Models for libraries and computational humanities  18 by volunteers on the Zooniverse platforms she explores the semantic history of the word ‘machine’ and the language relating to industrial accidents. C. Tasks and applications – what tasks are these models used for? To better understand the use cases for SLMs, we systematically examined the purposes that language models are used for – specifically, the tasks they perform within the context of humanities research. Below, we provide an overview of the most common use cases or tasks and determine which model is most appropriate (if applicable). The analysis provides a framework for LMs in humanities research. It establishes a correspondence between models and tasks. Supervised classification: this amounts to assigning labels to text, at either the document or the token level, from a pre-defined set of possible classes. Classification emerges as by far the most popular method. Small models, such as BERT (Devlin et al, 2019) (see more in appendix B), remain a popular and effective tool in this respect. While it depends on the complexity of the task and the number of annotations or training data required, SLMs and transfer learning remain a competitive paradigm. Bamman et al (2024) give an in-depth comparison between in-context learning with LLMs and traditional supervised classification with SLMs. The authors find that “prompt-based LLMs are competitive with traditional supervised models for established tasks but perform less well on novel tasks”. LLMs don’t always perform much better than their smaller peers. SLMs remain a critical tool for humanities research. However, the authors also observe that “LLMs can assist sensemaking by acting as an intermediary input to formal theory testing”. Instead of applying LLMs to classification problems for which we already have decent solutions, we should creatively repurpose them to explore novel investigative and interpretive ways of inquiry. LLMs can help us understand the category constructs and concepts we aim to capture when labelling data. They can support exploratory data analysis by suggesting possible features or patterns (that can later be interrogated through formal methods). Text generation for model introspection: the text generated by LMs is often useful as an object for humanistic inquiry. We can inspect how training data influences the models, as well as the behaviour of the model itself. D’Souza and Mimno (2023) investigate poetry memorisation using various smaller (GPT-2, Pythia 12B) and larger language models (PaLM, ChatGPT). They examine if these models can recall existing poems and explore the most effective retrieval models. Focusing on the top-performing model, ChatGPT, they analyse which characteristics of poets most strongly correlate with memorisation. Walsh et al (2024) probe the poetic style of ChatGPT, proposing and evaluating prompt templates for producing poetry then comparing the synthetically generated poems to ones in collections such as the Poetry Foundation and the Academy of American Poets. Text generation for corpus exploration: in if the sources could talk Garcia and Weilback develop and evaluate a RAG system for historical research. Their paper is an early example of conversational AI in the context of historical research. The authors assess a wide variety of language models (from smaller 7B models such as XGen and Beluga to GPT-3.5) and the extent to which these models support researchers in analysing tailored collections that include (1) primary materials (2) expert-authored secondary texts and (3) integrated sets comprising both. Their paper demonstrates that LLMs can deliver conversational support for research with a level of accuracy suitable for academic use. Although SLMs don’t yet match ChatGPT's performance across all areas, they do provide a viable, privacy-preserving option for handling question-answering tasks involving sensitive data. Modelling meaning with embeddings (see appendix B): embeddings are numerical representations of text, also referred to as vectors. As these embeddings capture aspects of meaning as a numerical they provide interesting material for computational analysis. Small Language Models for libraries and computational humanities  19 Embeddings extracted from historical LMs are especially useful for studying semantic change, allowing researchers to compute semantic shifts by comparing the embeddings of the same word at different points in time (for example, uses of ‘awful’ in the early versus the late 20th century). Mostly, researchers rely on cosine similarity to compute vector similarity. This measure is based on the angle between two vectors – i.e. if two vectors point in the same direction they will have a small angle between them and therefore a high (cosine) similarity. Zichert and Wüthrich (2024) apply semantic change detection to the concept of virtual particles, which in physics are understood as peculiar, short-lived particles that help explain how matter and forces interact. By studying the changing embedding of the word ‘virtual’ the authors show that, while its dominant meaning stabilises, it also gathers new, related ones. Self-supervised learning: this use case comprises domain adaptation through pre-training (using a language modelling objective). For more detailed explanations see the literature review section, the technical overview (appendix A) and the glossary (appendix B). Summary: why it matters… The analyses of CHR proceedings showed that SLMs remain a popular and critical tool, especially because of their adaptability to specific domains and tasks. Methodologically, researchers frequently rely on classification to analyse their data, using specific data and labels to express and find the concepts they investigate. For these reasons, SLMs and transfer learning will remain relevant in the years to come, as they offer adaptable and powerful tools that are computationally less demanding than the alternatives. Undoubtedly, the picture we sketched in this section will change over time. And as we also point out below, we anticipate that the field will move to a more mixed landscape, with hybrid workflows. LLMs undoubtedly have much to offer to computational humanities – from generating synthetic training data and qualitative data analysis to sense-making. We think of LLMs as an addition to the humanities toolbox, not necessarily as a replacement for SLMs. Small Language Models for libraries and computational humanities  20 Appendix A – Some more background to language models Historical overview This section sketches the history of LMs in broad brushstrokes. We introduce basic terminology and outline a framework that supports analysis of the literature, signposts to activities and offers some guidance. We have provided additional explanations in the glossary for words highlighted in bold. Language modelling Language modelling is the task of predicting the next (or missing) word. A language model (LM) calculates how likely a word is to follow in a given sequence. For example, in the sentence "predicting the future is hard, but not ..." the model would assign a higher probability to "impossible" than to "aardvark." More formally, language modelling assigns a probability to each possible word in the model’s vocabulary (Jurafsky and Martin, 2025). Pre-trained language models and transfer learning Originally, LMs seemed of little use, contributing to very few applications. During the 2010s LMs increasingly functioned as the backbone for almost any NLP application. Papers demonstrated that when learning to predict the next word these models absorb various types of information. LMs compress information in the data that can be leveraged for downstream tasks. This insight ushered in the paradigm of “transfer learning” (Howard and Ruder, 2018). Transfer learning encompasses two phases: pre-training and fine-tuning. Most current LMs are deep-learning models. Creating such models requires immense amounts of training data. While structured and labelled data remains scarce, unstructured data (i.e. text) is abundantly available. By teaching models to predict the next (or missing) word in a sequence, such massive unstructured information could be exploited to build better LMs. Pre-training involves iterating over large amounts of text data, word by word, and (at each step) asking the model to guess the next word, incrementally improving the ability to predict. During this process the model absorbs valuable linguistic patterns, ranging from syntactic regularities and semantic nuances to factual knowledge (and, more problematically, biases and stereotypes). The result of this operation is referred to as a ‘pre-trained’, ‘base’ or ‘foundation’ model. At this stage, the model only predicts words given an input (or generates texts by recursively repeating the token prediction and the addition to the input). Fine-tuning The breakthrough brought about by transfer learning was the development of techniques to fine-tune these base models and teach them to perform other tasks, based on labelled examples. For example, a base LM can be finetuned for sentiment mining, by fine-tuning it on sentences with sentiment labels. The general knowledge of language and the world is transferred to a novel task (namely sentiment detection). This machine-learning recipe has generally resulted in better performing, more adaptable NLP applications. Large language models and in-context learning GPT-3 (announced in May 2022) and ChatGPT (November 2022) pushed large and generative models to the centre of AI research. The GenAI paradigm shift was driven by the increasing scale of the models, the vastness of the pre-training data and the effectiveness of post-training methods. Small Language Models for libraries and computational humanities  21 Scaling up: Recently, models have increased dramatically in size. GPT-2 contained 1.5 billion parameters (or weights) whereas GPT-3 upped the ante to 175 billion. As the number of parameters grew, the scale of the training data also expanded drastically (Brown et al, 2020). Improved post-training was another key advancement. Supervised fine-tuning, which we have already discussed concerning pre-trained language models, is one example. Another breakthrough was the application of reinforcement learning with human feedback, which significantly enhanced the model's ability to follow instructions and align with specific expectations (ie, producing helpful answers and avoiding harmful content). These models are typically released in two versions: a base and an instruct model. The base model is pre-trained solely with a language modelling objective. These models can generate text but cannot be used in a conversational setting. For the latter, you must use the post-trained models, which are generally released with an ‘instruct’ suffix. For example, the Llama 3 model with 8 billion parameters is available on machine-learning platform Hugging Face as Meta-Llama-3-8B and Meta-Llama-3-8B-Instruct. In-context learning: With the increases in size, LLMs exhibited capabilities that were only faintly present in their smaller predecessors (Schaeffer et al, 2023). The main shift was from task-specific models to general problem solvers. These LLMs can be applied to a multitude of tasks without additional fine-tuning. They can adapt their behaviour and outputs to instructions or prompts written in natural language. A key concept in this regard is in-context learning. The model learns to perform a new task without changing its structure (more technically, without updating its weights, which is the case when fine-tuning). Simply describing the task is often sufficient (zero-shot learning), and if isn’t, providing one example (one-shot learning) or a few more examples (few-shot learning) as part of the prompt will often lead to improved performance. Small Language Models for libraries and computational humanities  22 Appendix B – Glossary Note that this was composed with the aid of ChatGPT and checked by the author for accuracy. BERT (Bidirectional encoder representations from transformers) A pre-trained language model developed by Google that reads text in both directions (left-to-right and right-to-left) to understand context better. BERT is widely used in natural language processing tasks such as question answering and sentence classification. Decoder model A neural network component that takes encoded information (usually in the form of numerical vectors) and generates output, such as translating encoded text into human language. Deep learning A subfield of machine learning that involves neural networks with many layers (hence ‘deep’). These models can automatically learn complex features from raw data, such as identifying faces in images or understanding the sentiment in text. Downstream task A specific application or objective that a pre-trained or fine-tuned machine-learning model is designed to perform, such as text classification or named entity recognition. These tasks follow initial training or pre-training phases. Embedding A numerical representation of data – such as words, sentences or images – captured in a fixed-size vector. Embeddings preserve semantic relationships and meaning. Encoder model The counterpart to the decoder model (above), an encoder transforms input data into a compressed, abstract representation. For example, in a language model, it might convert a sentence into a vector or embedding that captures its meaning, ready for further processing by a decoder or classifier. Epoch One full pass through the entire training dataset during model training. Multiple epochs are often necessary for the model to gradually learn patterns and minimise errors. Fine tuning A process in which a pre-trained model is trained again on a more specific or smaller dataset related to a target task. Fine-tuning enables the model to adapt its general knowledge to specialised contexts, and often improves performance without starting again from scratch. Generative model A model that can produce new data resembling its training data. These models can generate realistic text, images and audio and they are used in applications such as chatbots, art creation and data augmentation. Graphics processing unit (GPU) A specialised hardware component originally designed for rendering graphics, now widely used for accelerating machine-learning tasks, especially deep learning. GPUs enable faster training and inference. In-context learning Small Language Models for libraries and computational humanities  23 A technique where a language model learns to perform tasks by observing instructions and/or examples within a prompt, without updating its underlying parameters. The model draws on patterns from the context to infer the correct output, as seen in few-shot or zero-shot learning. Model parameter Internal values within a machine-learning model that are adjusted during training to reduce errors and improve predictions. Examples include weights in neural networks that determine how input signals are combined and transformed. Natural language processing (NLP) An area of artificial intelligence focused on enabling computers to interpret, generate, and respond to human language. Applications include machine translation, speech recognition and text summarisation. Post-training Any adjustments or processes applied to a model after it has been trained, such as pruning, quantisation or safety filtering. These steps may improve efficiency, reduce bias or make the model more suitable for deployment. Broadly speaking, post-training comprises three different strategies: • Reinforcement learning, where models are guided through reward-based feedback to align with desired behaviours • Fine-tuning, which involves further training on specialised datasets to adapt the model to specific tasks or domains • Test-time scaling, a method that enhances model performance by leveraging external tools or context during inference, leaving the model's parameters untouched Pre-training The initial phase of model training on a large, general-purpose dataset. Pre-training allows the model to learn fundamental patterns (such as grammar and structure) before being fine-tuned for a specific application. Quantisation Quantisation reduces the precision of the numbers that represent the parameters of a machine learning model. This makes the model smaller, faster and more efficient to run on hardware with limited resources, while trying to keep accuracy as close as possible to the original. Recurrent neural network (RNN) A type of neural network designed for processing sequences of data, such as text or time series. RNNs maintain a memory of previous inputs, allowing them to model temporal dependencies. They are foundational in early NLP models, although often replaced by transformers today. Reinforcement learning A machine-learning approach where an agent learns to make decisions by interacting with an environment, receiving rewards for beneficial actions. It is commonly used in areas like robotics, game playing and autonomous systems. Sentiment mining The task of identifying and categorising the emotional tone expressed in text, such as whether a product review is positive, negative or neutral. Often used in market analysis and social media monitoring. Structured and unstructured data Structured data is highly organised and fits neatly into tables, whereas unstructured data lacks a consistent format and includes content like text, images and audio. Small Language Models for libraries and computational humanities  24 Stochastic A term used to describe processes that involve randomness or probability. In machine learning, stochastic methods introduce random variation to help models avoid getting stuck in poor solutions. Synthetic data Artificially generated data that mimics real-world data. It is used to supplement training datasets when real data is scarce, expensive or sensitive. Techniques include simulations, random generation and use of generative models. Temperature A parameter that controls the randomness of a model’s output when generating text. A lower temperature (for example, 0.2) makes the model more confident and conservative, often producing repetitive or safe responses. A higher temperature (such as 0.8 or above) increases creativity and diversity in the output, but may lead to less coherent results. Training The process by which a machine-learning model learns from data by adjusting its parameters to reduce error. This involves feeding the model examples and comparing its predictions to the correct answers. The trained model can then be used to predict outcomes for new inputs. Training data The dataset used to teach a model during training. In the case of supervised learning it includes input-output pairs, allowing the model to learn the relationship between them (for example, text and its translation). For language modelling, the training data consists of text. Transformer A neural network architecture introduced in 2017 that has become the foundation for most modern NLP models. Transformers use self-attention mechanisms to process input data in parallel and capture relationships between all parts of a sequence, resulting in superior performance on many tasks. Vector A mathematical list of numbers that represent data in a format a machine-learning model can process. For example, a word might be represented as a vector that captures its meaning based on context. Vector space model A method for representing text or other data in a high-dimensional space where the distance between vectors reflects similarity. Often used in information retrieval and document comparison. Small Language Models for libraries and computational humanities  25 Appendix C – bibliography Abdin M, Aneja J, Awadalla H, et al. (2024) Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. Accessed 25 February 2025. Available at: http://arxiv.org/abs/2404.14219 Ardanuy MC, Nanni F, Beelen K, et al. (2020) Living Machines: A study of atypical animacy (arXiv preprint). https://arxiv.org/abs/2005.11140 Arevalo EM and Fonteyn L (2021) MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950). https://aclanthology.org/2021.nlp4dh-1.4/ Bamman D, Chang KK, Lucy L, et al. (2024) On Classification with Large Language Models in Cultural Analytics. https://ceur-ws.org/Vol-3834/paper119.pdf Beelen K and Strien D van (2022) Metadata Might Make Language Models Better. http://arxiv.org/abs/2211.10086 Beelen K, Lawrence J, Wilson DCS, et al. (2022) Bias and representativeness in digitized newspaper collections: Introducing the environmental scan. Digital Scholarship in the Humanities: fqac037. https://academic.oup.com/dsh/article/38/1/1/6644524 Bjerring-Hansen J, Al-Laith A, Hershcovich D, et al. (2024) Literary Time Travel: Distinguishing Past and Contemporary Worlds in Danish and Norwegian Fiction. https://ceur-ws.org/Vol-3834/paper19.pdf Brown TB, Mann B, Ryder N, et al. (2020) Language Models are Few-Shot Learners. http://arxiv.org/abs/2005.14165 Chang KK, Cramer M, Soni S, et al. (2023) Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4. http://arxiv.org/abs/2305.00118 Coll Ardanuy M, Beavan D, Beelen K, et al. (2021) Dataset for Toponym Resolution in Nineteenth-Century English Newspapers. British Library. https://bl.iro.bl.uk/concern/datasets/f3686eb9-4227-45cb-9acb-0453d35e6a03 Devlin J, Chang M-W, Lee K, et al. (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805 D’Souza L and Mimno D (2023) The Chatbot and the Canon: Poetry Memorization in LLMs. https://ceurws.org/Vol-3558/paper5712.pdf Garcia GG and Weilbach C (2023) If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History. https://arxiv.org/abs/2310.10808 Hagen T, Jannidis F and Witt A (2025) EncycNet: A Knowledge Graph of Historical German Encyclopedias. Zenodo. Accessed 14 April 2025. https://zenodo.org/doi/10.5281/zenodo Hosseini K, Beelen K, Colavizza G, et al. (2021) Neural Language Models for Nineteenth-Century English Accessed 14 April 2025. http://arxiv.org/abs/2105.11321 Howard J and Ruder S (2018) Universal Language Model Fine-tuning for Text Classification. Accessed 24 April 2025. http://arxiv.org/abs/1801.06146 Jo ES and Gebru T (2020) Lessons from archives: strategies for collecting sociocultural data in machine learning. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Barcelona Spain, 27 January 2020, pp. 306–316. ACM. Accessed 28 April 2025. https://dl.acm.org/doi/10.1145/3351095.3372829