Text Digitization

99 Text Digitization Daniel Stökl Ben Ezra https://orcid.org/0000-0001-5668-493X Abstract Text digitization describes the conversion of digital image data of inscribed objects into machine-readable texts.* Keywords HTR, OCR, Layout Analysis, Handwriting, Machine Learning, Neuron Networks 1. Introduction Let us define text digitization as the conversion of digital images of inscribed objects of any kind (manuscripts, inscriptions, cuneiform tablets, prints, etc.) into machine-readable texts. Databases with full-text search options in retroactively digitized prints such as JSTOR (since 1994) or Google Books (since 2004) have fundamentally changed the way research is carried out in all academic disciplines. In the last ten years, progress in automatic document analysis, especially in machine learning, has revolutionized the possibilities for researchers to analyze not only difficult prints of the most important cultural texts, but even historical manuscripts, with computerized means. Transcription of the letters is not the only level of analysis. Text goes beyond a sequence of letters. Gerard Genette (1982) has highlighted the importance of layout and non-main text sections for the (preliminary) understanding of texts. Layout contains critical information, such as the distinction between title and main text, main text and notes, speakers in dramas, verses in poetry, or text connections in translations or commentaries. The choice of and changes in writing style or typeface, width, register (e. g. normal, italic, slanted), weight, color, and alphabets are also essential information carriers that can significantly deepen the depth of analysis of an analyzed text beyond the simple letter sequence (Beinert 2021). In addition, machine paleography, layout analysis, and codicology are used to evaluate these subtle differences for network analysis, dating, and localization of individual objects. This big data provides the traditional auxiliary sciences a completely new meaning. * This chapter, including quotations in foreign languages, was translated from German by Brandon Watson. DanielStöklBenEzra:TextDigitization,in:ChristopherA.NunnandFrederikevanOorschot (eds.):CompendiumofComputationalTheology1.IntroducingDigitalHumanitiestoTheology, Heidelberg:heiBOOKS,2024,S.99–116.DOI:https://doi.org/10.11588/heibooks.1521.c21942 Daniel Stökl Ben Ezra100 After a brief introduction about the encoding of image and text, neural networks, and existing programs, this chapter treats layout analysis and reading order, computer paleography and text recognition. The analysis is in constant engagement with the discipline known in computer science as “image processing” (Maier et al. 2020). 2. ImageEncoding Computers can only distinguish between the values 0 and 1. Both texts and images must initially be represented as sequences of 0s and 1s. Usually, the image is then transferred with a grid into a table/matrix with rows and columns, in which each cell contains a value for one pixel. High-resolution images have more pixels for the same object surface than low-resolution images. Black and white images are the simplest image format, only knowing 0 or 1 for each pixel, such as foreground (ink) or background (paper). This format leads to the familiar stair steps (Fig. 1), especially at low resolution or high magnification. For more nuanced image, one can use grayscale images that allow intermediate levels. 0 still stands for black, but instead of 1, 255 is now used for white, and all numbers in between denote gray values, depending on whether they are closer to white or black. These are so-called 8-bit images, i. e., a combination of eight memory cells (bits) for each pixel (256 = 28). For finer gradation, one can also use more extensive grayscale and then work with values from 0 to 65,555 (16-bit – 216) or more. The lowest value is used for black and the highest value for white, and the values in between represent the possible shades of gray. Fig. 1 AbrahamLincolnasagrayscaleimagewiththevaluesonthescaleof0 – 255 Text Digitization 101 Color images use a combination of three superimposed grayscale images with an 8-bit scale of 0 – 255. The color composition (e. g., red-green-blue = RGB) produces the mixed colors for eyes, e. g., violet, brown, orange, or pink. Instead of RGB, other color channels can also be used, e. g., cyan, magenta, and yellow, as in color printers. And, as with grayscale images, a much finer 16-bit scale can be selected instead of an 8-bit scale. The greater the height and width, resolution or color scale, the more memory an image requires. Image data is often compressed to save space or increase processing speed. Each camera or scanner manufacturer has its own proprietary format (Raw). When exported, this format is converted into TIFF, PNG, or JPEG image files. Depending on the compression, Tiff and Png require considerably more storage space than JPEG image files (extension jpg), which are created using a compression algorithm that accepts more loss of information to take up less storage space. This process creates artifacts, which can be easily recognized at high magnification the image looking like bathroom tiles. As long as the image resolution is good (at least 30 pixels for an ‘a’ or ‘א,’ much more for more complex scripts such as Chinese), jpgs are sufficient for layout analysis and text recognition. Automatic paleography or writer identification achieves better results with tiff and png files. 3. TextCoding In a computer, a text is also stored as sequences of 0s and 1s. In the past, when memory was more expensive, 8 bits were reserved for all variants together for each character, so that computers only knew 256 different values (code points), which could only represent a selection of either Latin, Greek, Cyrillic, or Hebrew, depending on the linguistic or geographical workplace, and was known as the Extended ASCII table. An additional code in the text file indicated which alphabets were meant by the raw numbers in the file. It was thereby possible to work with the local alphabets in Germany, Bulgaria, Israel, or Saudi Arabia, but never with all scripts at the same time. More complex scripts with a total number of characters greater than 256, such as Chinese, were impossible. Furthermore, if the encoding schema was unknown, the only solution was to try out all encodings until the text was displayed legibly. This limitation to 256 different characters was a major challenge for philological work in theology, where Syriac, Arabic, Armenian, Georgian, or Coptic – and sometimes also Akkadian, Egyptian, Ethiopian or Sanskrit – were often used in addition to Greek and Hebrew. The introduction of the Unicode standard in 1991 has progressed towards solving this problem. Like grayscale images with finer nuances, the memory reserved for each character was initially doubled from 8 bits to 16 bits, allowing 216 = 65,536 different code points in the table. In 2022, a total of 161 fonts could be defined in a single table using this encoding, also known as UTF-8. Even though the introduction of the Daniel Stökl Ben Ezra102 Unicode was a major step forward, difficulties for the digital modelling of historical scripts remain, e. g., there is still no Unicode for the Babylonian vocalization of Hebrew texts, and Egyptian or Akkadian have only been partially standardized. Humanities scholars should be familiarized with the intricacies of Unicode because the basic questions of script encoding that need to be solved are tricky. The present chapter will return to further questions, like reading order in bidirectional texts that mix Hebrew and Latin, or character combination coding, in the text recognition section. 4. NeuralNetworks The rapid progress in automatic document analysis in recent years has five main interdependent causes: Hardware (memory, speed), software (neural networks), training data volumes, mass digitization and open-source policies. Processors have become much faster and can process much larger amounts of data simultaneously thanks to increased memory. Hard disk storage and internet data transmission (fiber optics) are also cheaper and of much better quality than 10 years ago. Mass digitization projects of culturally significant manuscript collections, archives, and libraries have led to a flood of image data. The interest of large corporations in processing large amounts of written and oral text (Google Books, YouTube, Netflix, Zoom) has not only improved existing algorithms but also developed new ones. Some of these algorithms are freely available to the public in open-source packages of the most important programming languages (e. g., pytorch from Facebook, TensorFlow from Google). There are research projects that have published their training data under open licenses, thus enabling others to use them to develop or optimize new algorithms. Different forms of artificial neural networks are used for almost all stages of automatic document analysis. The basic principle has been known since Rosenblatt in 1958, but it was the above-mentioned constellation of simultaneous progress in hardware, software, data, and open source that led to their success (starting with Jürgen Schmidhuber and Yann LeCun’s works in the early 1990s). The common principle is a very complex formula with thousands, millions, or even billions of parameters that are optimized by the computer in a learning process called training. The result of a trained network architecture is called a model because it mathematically models the problem (Fig. 2). To a certain extent, artificial neural networks imitate the way brains work. The three most relevant network types at present are Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) and Transformers. The most important common principle is the inclusion of context for each data point. CNNs are particularly interesting for images because a data point (i. e., a pixel) is considered in the context of a rectangle. For example, the computer can learn abstract concepts such as curves of Text Digitization 103 different curvature and lines at different angles and orientations and combinations. RNNs are interesting for sequences such as audio recordings or texts, as they are flexible in learning how much context to include for a certain phenomenon. For longer sequences, Long-Short Term Memory Neural Networks (LSTM), are often used. In deep learning, several layers of CNNs and/or RNNs are combined, resulting in complex architectures that require large amounts of data and often a lot of time to train. At the same time, these combinations also deliver excellent results, both for layout analysis and for transcription as well as many other tasks like classification. Transformers have become commonplace through recent Large Language Models (LLMs), such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer). With the right configuration and an optimal learning process, they can learn to perform layout analysis simultaneously with transcription, recognize proper names (Named Entity Recognition), translate, or summarize the resulting texts or even answer questions. When training, a distinction is made between supervised, self-supervised, and unsupervised training processes. In the supervised training process, the computer is shown questions with the corresponding answers and tries to optimize the parameters of the formula to arrive at the answer from the question step by step. Question/ answer pairs depend on the learning objective and can be very different depending on the task: (1) the image of a line of text and the corresponding transcription; (2) the image of a manuscript page and the corresponding polygons of the layout; or (3) the image of a book page and the corresponding print type. Both question and answer are represented as a number (scalar), number sequence (vector), matrix or tensor etc., since computers know nothing else. At the beginning, all parameters are often initialized randomly. After each learning step (question/answer calculation), the distance of the calculated answer to the correct one is measured and the parameters are adjusted so that the next time the computer is confronted with the same or similar question, the calculated answer is closer to the correct one. If the distance gradually diminishes (and it is not always the case), then the model is said to converge. At regular intervals, the computed formula is subjected to a test in which it is presented with question/answer pairs without the computer learning from them, i. e., without changing the formula, but only determining the current precision of the current model. The specialist tries to formulate the network architecture so that that it can achieve the best models with as little training material as possible and in the fastest possible computing time. Training is usually terminated when the user determines from the test results that the computer is not improving any further. Finally, the model that delivered the best results in the comparison is saved. The training process is not uniform. Consider the following illustration: a spaceship is supposed to fly as far as possible through a complex labyrinth of caves but is only allowed to make turns at a certain angle and must then fly a certain distance in this direction. If the selected distance is too short, the spaceship will crawl through the large opening space without finding one of the entrances to the labyrinth. If the Daniel Stökl Ben Ezra104 distance is too long, the spaceship will not be able to pass through tight turns. Angle and distance are among the hyperparameters of the training process, perhaps best compared to the learning rate. In the self-supervised training process, the computer automatically calculates the answers from the questions. For example, the computer is shown many images of lines of handwriting, some of which are blacked out, and is asked to suggest a substitute image of what the blacked-out section of the line might have looked like. The model is optimized by analyzing the difference between the computer’s suggestion and the original image. In this case, the computer gradually learns the principle of which pixel clusters are usually located between which other pixel clusters. Selfsupervised training is sometimes used as pre-training before supervised training because it allows basic principles to be learned beforehand and reduces the amount of training data with manually generated responses. One potential danger is overfitting. We can compare this to a student who memorizes the answers to the practice questions without understanding the underlying principle. The student can answer the practice questions almost perfectly but is unable to solve unseen questions. The same can happen, for example, if the training corpus is not adapted to the network architecture (too little training material for a question that is too complex) or if the learning rate at which the computer tries to adapt the parameters after each learning process is set too high or too low. Of course, we have only been able to describe a few hyperparameters here. Fig. 2 Examplediagramofaneuralnetworkwithaone-dimensionalinputlayer(withfour variables)thatleadsthroughacomplexnetworktoanoutputlayerwithtwovariables.Eachofthe linesisamathematicaloperation,theparametersofwhichmustbeoptimized. Text Digitization 105 5. ExistingPrograms Until recently, most companies and individual researchers interested in OCR worked with commercial programs like ABBYY Finereader, which was very successful with modern English or German texts but could not handle handwritten material or rarer printed texts such as the Syriac alphabets, for which the market seemed too small. Google’s widely used open-source program Tesseract is only recommended for OCR but not for HTR.1 Currently, there are several successful programs used for mass digitization of handwritten material. Since 2016, Transkribus has enabled the automatic layout analysis and transcription of written objects, the manual correction of layout analysis and recognition, as well as the training of own transcription models based on the entered data with excellent results via a complex JAVA app or a simplified web app (Kahle et al. 2017, Mühlberger et al. 2019). The program, originally developed in several European research projects, was commercialized in 2019 in the form of a European cooperative.2 Currently, users pay per page for automatic layout analysis and/or automatic text recognition. The platform and trained models are therefore closed-source. Other commercial programs include Ocelus and Calfa (Vidal Gorène 2021a). In open-source, OCRopus/ocropy, developed by Thomas Breuel (2008), was a decisive step forward. Although, as the name suggests, it was only developed for OCR, our Paris team, with the help of Marcus Liwicki, has also been using it for handwriting recognition since 2015. However, programming knowledge was a prerequisite for use. There was only a very rudimentary way to enter transcriptions, only for even lines and only very simple segmentation. Since 2018, eScriptorium has been developed around Benjamin Kiessling’s Kraken (Kiessling et al. 2019, Stokes et al. 2021). It is currently the only open-source program for handwriting analysis with an ergonomic user interface for layout and transcription correction as well as text alignment. It can be installed directly on Linux, Mac OS, and Windows computers using WSL (Windows Subsystem for Linux). If a team wants to collaborate on the same document(s), a server is required. A GPU with sufficient RAM is needed to train layout or transcription models. 1 FAQ: https://tesseract-ocr.github.io/tessdoc/FAQ.html#can-i-use-tesseract-for-handwriting-recogni tion (Accessed: 15 June 2024). 2 See https://readcoop.eu/a-short-history-of-transkribus-with-gunter-muhlberger (Accessed: 15 June 2024). Daniel Stökl Ben Ezra106 6. LayoutAnalysis Computerized layout analysis has two objectives. First, computerized analysis is – thus far – a necessary step before text recognition. Second, the layout contains essential information for the hierarchy, reading order (see below), different text types, differentiation between image and text, etc. Layout analysis is crucial in text comprehension even after text recognition. Previously, morphological operations were used for layout analysis to recognize different text blocks and lines. Currently, this is accomplished by neural network architectures (Fig. 3) that manage both the segmentation of regions and their division into types (column, header, marginalia, illustration, table, apparatus, etc.) as well as the recognition of lines and their division into types (main text line, interlinear line), and the writing direction (horizontal, vertical, upside down). A segmentation ontology determines which region and line types can be used for which phenomena. Incidentally, zones do not necessarily have to be text regions. Users can also use image segmentation to locate library stamps, coins, illustrations, etc. There are currently two different approaches to region segmentation. One approach uses principles for object recognition such as traffic lights or signs in self-driving cars (Clérice 2022). This approach works very well for text objects with only rectangular regions that have been digitized in a precise fashion, e. g. without tilting, sheering or rotation. However, problems quickly arise with more complex layouts, e. g., L-shaped regions, or with small rotations. The other approach is a pixel classifier (Kiessling 2020). All image pixels are assigned to one or more desired types of regions. The pixel cloud is then determined for each region type and one or more polygons are reconstructed. This approach better manages complex layouts or rotated digitized images but has difficulties assigning pixel groups of the same type that are very close to each other to the same polygon. The approach therefore tends to classify two closely spaced main text columns as a single zone. Line segmentation occurs simultaneously to or after region segmentation (Grüning 2017). There are also two approaches in line segmentation. Either a neural network is trained first to detect the baseline and writing direction of each line and then calculate a polygon that surrounds this baseline so that all ink traces of the characters in this line, including any dots and dashes above or below them, are included. Or the neural network is trained to recognize the line polygon directly and then derive the writing direction. If the training data is homogeneous and numerous enough, in kraken/eScriptorium, very complex segmentation models can be trained with 20 different region and line types (Stökl Ben Ezra 2022b). Training simple specific segmentation models is possible with just a few training pages. Text Digitization 107 7. ReadingOrder Different text types diverge in their reading order. There are very different ways to read texts, such as the sections of one or more newspaper pages, a critical edition, a bilingual edition, a table, a manuscript with a basic text or with commentaries, a letter, or postcards. Poetic texts are often written stychographically looking like two columns of a prose text. Reading order analysis is closely related to layout analysis. In kraken, layout analysis can currently be trained in a version still under development, the integration of which into eScriptorium will take a few more months to design and implement the necessary ergonomic user environment (for the method, see Quirós 2022). Fig. 3 FullyautomatedlayoutsegmentationandtranscriptionoftheGenizahfragmentCambridge,T.-S.10J1216witheScriptorium,createdintheHTR4PGPproject. Daniel Stökl Ben Ezra114 Stökl Ben Ezra, D., Brown-DeVost, B., Dershowitz, N., Pechorin, A., & Kiessling, B. (2020). Transcription Alignment for Highly Fragmentary Historical Manuscripts. The Dead Sea Scrolls. In International Conference on Frontiers in Handwriting Recognition (pp. 361 – 366). Dortmund: IEEE Xplore. DOI: https://doi. org/10.1109/ICFHR2020.2020.00072 (Accessed: 15 June 2024). Stökl Ben Ezra, D., & Lapin, H. (in print). From HTR to Digital Critical Scholarly Edition. Reflexions on the Use of Machine Learning. Computational and Digital Humanities in the Sofer Mahir Project. In U. Henny-Krahmer et al. (Eds.), Machine Learning and Data Mining for Digital Scholarly Editions. Norderstedt: Books on Demand [= SIDE]. Stökl Ben Ezra, D., Lapin, H., Brown DeVost, B., & Jablonski, P. (2022a). HTR2CritEd. A Semi-Automatic Pipeline to Produce a Critical Digital Edition of Literary Texts with Multiple Witnesses out of Text Created through Handwritten Text Recognition. In Digital Humanities 2022. Responding to Asian Diversity (pp. 690 – 691). Tokyo: DH2022 Local Organizing Committee. URL: https://dh2022.dhii.asia/ dh2022bookofabsts.pdf (Accessed: 15 June 2024). Stökl Ben Ezra, D., Rustow, M., & Witty, D. (2022b). Segmentation Mode for Archival Documents with Highly Complex Layout. In Conference Documents anciens et reconnaissance automatique des écritures manuscrites. École national des chartes, Paris: YouTube. URL: https://www.youtube.com/watch?v=dE1XUXiuitU (7:07 – 7:30). (Accessed: 15 June 2024). Ströbel, Ph., Hodel, T., Fischer, A., Scius, A., Wolf, B., Janka, A., Widmer, J., Scheurer, P., & Volk, M. (2023). Bullingers Briefwechsel zugänglich machen. Stand der Handschriftenerkennung. In A. Busch, & P. Trilcke (Eds.), DHd 2023. Open Humanities, Open Culture (pp. 98 – 102). Belval/Trier: Zenodo. DOI: https://doi. org/10.5281/zenodo.7688631 (Accessed: 15 June 2024). Wolf, L., Littman, R., Mayer, N., German, T., Dershowitz, N., Shweka, R., & Choueka, Y. (2010). Identifying Join Candidates in the Cairo Genizah, International Journal of Computer Vision, 94(1), 118 – 135. LiteratureRelated Camps, J.-B., Vidal-Gorène, Ch., & Vernet, M. (2021). Handling Heavily Abbreviated Manuscripts. HTR Engines vs Text Normalisation Approaches. In E. H. Barney Smith & U. Pal (Eds.), Document Analysis and Recognition – ICDAR 2021 Workshops. ICDAR 2021 (pp. 306 – 316). Cham: Springer [= Lecture Notes in Computer Science, 12917]. DOI: https://doi.org/10.1007/978-3-030-86159-9_21 (Accessed: 15 June 2024). Chagué, A., & Thibault, C. (2023). I’m here to fight for ground truth. HTR-United, a solution towards a common for HTR training data. In Digital Humanities Text Digitization 115 2023. Collaboration as Opportunity. Graz: Zenodo. DOI: https://doi.org/10.5281/ zenodo.8107449 (Accessed: 15 June 2024). Clérice, Th. (2022). You Actually Look Twice At it (YALTAi). Using an object detection approach instead of region segmentation within the Kraken engine, Journal of Data Mining and Digital Humanities, 1 – 13. DOI: https://doi.org/10.48550/ arXiv.2207.11230 (Accessed: 15 June 2024). Perdiki, E. (2023). List of manuscripts containing John Chrysostom’s Homilies and the relevant manual transcriptions, 1(2). Zenodo. DOI: https://doi.org/10.5281/ zenodo.8102662 (Accessed: 15 June 2024). Popović, M., Dhali, M. A., & Schomaker, L. (2023). Artificial intelligence based writer identification generates new evidence for the unknown scribes of the Dead Sea Scrolls exemplified by the Great Isaiah Scroll (1QIsaa), PLoS ONE, 16(4. e0249769), 1 – 28. DOI: https://doi.org/10.1371/journal.pone.0249769 (Accessed: 15 June 2024). Vidal-Gorène, Ch., Dupin B., Decours-Perez A., & Riccioli T. (2021a). A Modular and Automated Annotation Platform for Handwritings. Evaluation on Under-Resourced Languages. In Lladós, J., Lopresti, D., Seiichi, U. (Eds.), Document Analysis and Rcognition. ICDAR 2021 (pp. 507 – 522). Lausanne: Springer. DOI: https:// doi.org/10.1007/978-3-030-86334-0_33 (Accessed: 15 June 2024). Vidal-Gorène, Ch., & Decours-Perez, A. (2021b) A Computational Approach of Armenian Paleography. In E. H. Barney Smith & U. Pal (Eds.), Document Analysis and Recognition – ICDAR 2021 Workshops. ICDAR 2021 (pp. 295 – 305). Cham: Springer [= Lecture Notes in Computer Science, 12917]. DOI: https://doi.org/10.1007/978-3030-86159-9_20 (Accessed: 15 June 2024). Wick, Ch., Reul, Ch., & Puppe, F. (2018). Comparison of OCR Accuracy on Early Printed Books using the Open Source Engines Calamari and OCRopus, Journal of Language Technology and Computational Linguistics, 33(1), 79 – 96. FigureCredits Fig. 1: Abraham Lincoln as a grey scale image. CC0. The image was probably originally created by Leon Harmon in 1971, who wanted to find out how much visual information an image could do without in order to still be recognisable. Fig. 2: Wikipedia © “BrunelloN” CC-BY-SA 4.0 Fig. 3: Screenshot Daniel Stökl Ben Ezra. Manuscript © CC-BY Cambridge University Library Daniel Stökl Ben Ezra116 Funded by the European Union (ERC, MiDRASH, Project No. 101071829). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.