Enhancing YOLO Models for Handwritten Text Recognition

Enhancing YOLO Models for Handwritten Text Recognition Nikita Lomov HSE University Moscow, Russia [email protected] Abstract—This paper is devoted to the development of computer vision models capable of solving problems of simultaneous detection and recognition of handwritten text. As a starting point, the YOLOv8 family of architectures for object detection is considered. We formulate line detection as different tasks depending on the text shape: straight lines as object detection, slanted lines as oriented bounding box (OBB) detection, and curved lines as instance segmentation. For each version of the model, a suitable pooling procedure is developed that extracts a feature description within a bounding box or a mask. For the instance segmentation problem, a modification of the segmentation mechanism is proposed that takes into account the features of lines as graphic objects and operates on a geometric principle. To recognize the formatting of handwritten text, in particular, to determine strikethrough and underlining, a transition to an extended alphabet is carried out with the prediction of two components—a symbol and its style—separately. The effectiveness of the developed methods is estimated on real original data—a set of diary pages of the Russian statesman Modest Andreevich Korf (1800-1876), which is a valuable historical source. All models successfully cope with the task and demonstrate a character error rate (CER) of about 3-4%, which makes the recognized text easily readable by a person. At the same time, the quality of recognition increases with increasing complexity of the model, which justifies the consideration of various variants of the problem. The code is available at https://github.com/nlomov/yolo-htr. I. INTRODUCTION In recent years, the widespread use of digital technologies and machine learning tools has had a significant impact on the nature of the work of researchers in the humanities. Not only has the digitalization of archives and document collections made it possible to access them remotely, but methods for intelligently analyzing the documents themselves, including searching and navigating them, summarizing and categorizing them, make it possible to cover previously unimaginable volumes of information. Handwriting recognition, aimed at converting a scanned or photographed document image into a textual transcription, is a fundamental challenge in this field. Several software tools offer complex handwriting recognition functionality, including Kraken OCR [1], Transkribus [2] and OCR4all [3]. They provide broad opportunities for solving problems associated with HTR, such as extracting, binarizing and normalizing text lines, layout analysis, collecting training samples and training your own models. Among the resources focused on recognizing documents in Russian, we can highlight Yandex Vision OCR [4], commercial system with access via API. Also, handwriting recognition systems based on multimodal language models are easy-to-use and offer enormous potential [5]. At the same time, even large datasets do not eliminate the problem of handwriting diversity, which makes it difficult to apply a pre-trained model to previously unseen writing styles. Even greater obstacles are associated with insufficient amounts of data for training in the case of non-trivial languages and alphabets other than Latin. For these reasons, the development of computer vision models that can train on small amounts of data, for example, handwriting samples of a particular person, still seems to be a relevant task. The study [6] shows that when tuning for a specific handwriting, the volume of training data of several hundred lines is insufficient, since stabilization of quality metrics is not achieved, so it is more appropriate to talk about thousands of lines. Thus, the task of developing handwriting recognition models that are both user-friendly and easy to train remains a pressing issue. The actual survey [7] on the problem considers two groups of approaches: up-to-line level, including word and line recognition, and beyond line-level, concerning paragraphand document-level challenges. However, the input data is rarely represented by images of lines, so using the methods in the first class requires extracting the strings from the image, and often doing significant preprocessing of those crops. It is natural to encapsulate all subtasks within a single model, which defines our choice in favor of page-level approaches that, owing to advances in computing power, have been attracting increasing research interest. As an additional task we consider the analysis of handwritten text formatting, in particular the highlighting of underlined and crossed-out fragments, interlinear insertions, and characteristic abbreviations. Formatting features help to trace the progress of work on the manuscript, compare its various versions, record what the author himself paid special attention to, and what he considered generally applicable. For example, such problems were addressed by researchers of the works of Lope de Vega [8] and Charles Dickens [9]. Our work is aimed at creating page models that combine search and recognition of lines of formatted handwritten text, for various formulations of the search problem—such as classical detection, oriented bounding box detection and instance segmentation. The main achievements of this work are as follows: ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 383 ---------------------------------------------------------------------------- •Three main architectures from the YOLOv8 family are modified in such a way as to support text recognition within the corresponding area. •We propose a new principle of instance segmentation, which works on a geometric principle and copes better with visually similar objects. •We demonstrate a method for recognizing text formatting that uses an extended alphabet of symbol-style pairs. II. RELATED WORK Models aimed at recognizing line-level images are quite well developed and varied in architecture. They can be fully convolutional [10], use multivariate recurrent connections [11], or be based on cutting-edge architectures like Visual Transformer [12]. There are a number of works in which a full-fledged text recognition system at the page level is assembled from several modules, one of which is the extraction of text lines, which allows for end-to-end inference, but not end-to-end training. In particular, the system described in [13] includes not only separate networks for searching and recognizing marginalia, and the second of them contains a learning module based on thin-plate spline for straightening words. The system presented in [14] goes down from the page level to the level of individual words, different neural network architectures are compared to solve subtasks. In [15], the authors combined YOLOv8-driven detector of text regions with a simple RNN-based handwritten text recognition model. Some models work at the paragraph level, assuming that lines of text are stacked neatly on top of each other and not too warped. To distinguish individual lines, vertical attention [10] or sequential vertical upsampling and horizontal downsampling to collapse the horizontal dimension [16] is used. It is worth noting that models that work directly at the page level are still relatively rare [17]–[19] and are quite difficult to train—for example, they usually involve pre-training on synthetic datasets. The first successful attempt to implement the end-to-end paragraph handwritten text recognition was undertaken in [20], where text scanning was carried out using the attention mechanism implemented by a multi-dimensional Long ShortTerm Memory (LSTM) network. In the Start, Follow, Read method [21] three networks are trained together at once: to search for the starts of lines, to track them with the formation of a straightened image of the line, and to recognize its text. One way to make the network end-to-end learnable is to connect the output of the detector to the input of the recognizer using some pooling procedure. In [22], a RoI Pooling operation is used, which consists of resampling the detected word from the original image by a grid of a fixed height. In [23], a similar operation is called Text Pooling and extracts crops of feature masks for lines of text, but not words. In [24], features are extracted within the rotated box, and an adversarial feature learning network is used to approximate the distributions of these features and the synthetic image’s ones. The [25] uses a spatial transformer network to detect words of non-handwritten text in natural scene images and allows a wider class of affine transformations during resampling. The tasks of detecting text elements—strings of words and letters (usually in the case of hieroglyphic writing)—can be combined with various recognition tasks. Thus, the work [26] demonstrates an example of simultaneous recognition of the letter itself and its box, for which the CTC loss is modified. In the work [27], the problems of detecting handwritten words, classifying them by types of named entities and direct text recognition are simultaneously solved. In [28], various types of text formatting, including strikethroughs, underlines, and multi-line text, are detected using a network with an architecture based on the Generative Adversarial Network (GAN). The tasks of formatting recognition are often accompanied by image restoration tasks. Thus, in the work [29] the word detection problem is solved, strikethrough words are presented as a separate class. Then the appearance of the word without strikethrough is restored by a combination of U-Net and Bi-LSTM networks. Search and removal of user marks, such as underlining, is implemented in the work [30] using line slope analysis and morphological operations. III. ARCHITECTURE DESIGN The work [31] proposed the YOLO-HTR architecture, the idea of which is to replace the YOLOv8 network encoder with an encoder from a network designed to recognize text lines. The features obtained by the Vertical Attention Network (VAN) [10] have proven themselves to be good for text recognition. The network encoder consists of 10 consecutive blocks, like the YOLOv8 network encoder, therefore, as in the YOLOv8 network, the outputs of the 5th, 7th and 10th blocks are sent to the head part, which is directly responsible for the regression of the box parameters. Next, the features from the last layer of the encoder were masked by boxes and freed from vertical dimension using a special box pooling procedure. Image Encoder (VAN) Region Decoder (YOLOv8 Head) Region Pooling Text Decoder (LSTM + FC) image feats region feats image feats region params & probs Lreg +Lcls Lctc char probs Fig. 1. General scheme of the YOLO-HTR architecture. The region can be a bounding box, an oriented bounding box, or a bounding box with a mask. In the first two cases, Lreg is equal to Lbox, in the last case—to Lbox +Lseg. Hidden layer size in LSTM in Text Decoder is 512. We will follow the general scheme of this approach, shown in Figure 1. Note that in the original version of YOLO-HTR only a network for classical object detection with straight boxes was considered. We will develop networks for oriented bounding boxes’ detection and instance segmentation, too, with heads taken from the corresponding YOLO variants. We also need to modify the connection between the encoder and the region decoder in order to better handle lines—objects that are usually wide and low. ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 384 ---------------------------------------------------------------------------- Indeed, it should be noted that the text recognition encoder is local and pays attention to a small neighborhood of the pixel, sufficient to recognize an individual character and its neighbors. At the same time, for successful detection of the entire line, the network must “see” it in full width with some neighborhood, and therefore must have a sufficiently wide receptive field. Let us estimate its size for all layers of the VAN encoder and three detection blocks—in small (S), medium (M) and large (L) scales—with a “naive” replacement of the encoder. TABLE I. THE SIZES OF THE RECEPTIVE FIELD WITH A ”NAIVE” ENCODER REPLACEMENT Block Scaled Original Block Scaled Original Encoder 17x7 7x7 Encoder 832 x22 256 x 352 Encoder 28x13 16 x13 Encoder 938 x28 304 x 448 Encoder 38x11 32 x22 Encoder 10 44 x34 352 x 544 Encoder 48x10 64 x40 Encoder 514 x10 112 x80 Detector S80 x 104 640 x 832 Encoder 620 x10 160 x 160 Detector M53 x65 848 x 1040 Encoder 726 x16 208 x 256 Detector L39 x45 1248 x 1440 Table I presents the receptive field sizes in the downsampled resolution (columns ”Scaled”) and in the original one (”Original”). The results show that the maximum line that the detector can capture is 1248 pixels wide. This may not be enough to describe long lines, especially with random scaling during augmentation. Note that the original YOLOv8 network has field sizes of Detector S/M/L in original pixels equal to 1728, 1728, and 2368, the same in width and height. To expand the network’s field of view, we add separately to each of the layers read out (5th, 7th and 10th) the corresponding layer from the YOLO encoder. As a result, we increase the field sizes of Detector S/M/L to 1248x1440, 1648x1840 and 2080x2272, which is quite enough for training on large images of about 2000 pixels. W H 512 W 16 H 16 AvgPool (1,2) l 2 256 W 8 H 8 C2f 1 256 W 16 H 16 AvgPool (1,2) o 6 6 256 W 16 H 16 C2f 2 16 W H CB 1 32 W 2 H CB 2 64 W 4 H 2 CB 3 128 W 8 H 4 CB 4 256 W 8 H 8 CB 5 256 W 8 H 16 CB 6 256 DSCB 1 256 DSCB 2 256 DSCB 3 512 W 8 H 16 DSCB 4 512 W 32 H 32 SPPF Fig. 2. Encoder of YOLO-HTR+ architecture. The arrows pointing down lead to the region (Box/OBB/Mask) decoder, the right arrow leads to the region pooling. A detailed diagram of the new YOLO-HTR+ network encoder is shown in Figure 2. Minor modifications were made to the VAN encoder in terms of layer sizes and downsampling rates to enable more successful processing of closely spaced lines based on the results of previous work on YOLO-HTR. Also, the decoder structure in the form of a combination of LSTM and a fully connected (FC) layer was taken from the page-level, rather than the line-level, version of VAN, since it was recognized more suitable for processing complex handwriting. IV. REGION POOLING PROCEDURE The aim of the region pooling procedure, which does not have trainable parameters, is to preserve the features of only those points that fall within the predicted box and to eliminate the vertical dimension in preparation for text recognition using CTC loss. The specific type of region pooling operation depends on the region representation—as a straight bounding box, an oriented bounding box, or a straight box with a mask. The input to the procedure is a set of boxes b={bi}m i=1 and a feature map F∈RH×W×n, at the output we have a tensor G∈Rm×W×n. Since the procedure is carried out independently by features and boxes, it is sufficient to examine the map of a single feature Ft∈RH×Wand the single box b, the obtained result F t∈R1×Wwill be a slice of G. A. Straight Boxes For straight boxes, the region pooling procedure is described in [31]. Assuming that the box coordinates b=(x, y, w, h) are specified in the scale of the feature map, for each of its H×Wcells we determine the fraction of the cell that falls within the box: wx(j)=1−clip x−w 2−j−1−clip j−x+w 2, j=1,...,W, wy(i)=1−clip y−h 2−i−1−clip i−y+h 2, i=1,...,H, where clip(a) = min(max(a, 0),1). Considering that wx∈R1×W,wy∈RH×1, the map of the t-th feature Ftmasked by the box, can be represented in matrix form F t=(wywT x)Ft. For aggregation, we average the values inside the box vertically, noting that the sum of wyis equal to hin the case when the box fits completely into the image: F t=1 h[1, ..., 1] ×F t. Developing analogues of the pooling procedure for detection tasks in other formulations will provide a more accurate description of line shapes. This, in turn, will enable the extraction of more relevant text features, leading to improved text recognition quality. B. Rotated Boxes In this case, the box is augmented by an angle α:b= (x, y, w, h, α). We define the centers of all cells in the feature map Ft: ¯xj=j−0.5,¯yi=i−0.5, C={(¯xi,¯yj)},j=1,...,W,i=1,...,H. ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 385 ---------------------------------------------------------------------------- When both the box itself and the points of the set Care rotated by an angle −α: x y=Mx βy,¯x ij ¯y ij =M¯xj β¯yi, M=cos αsin α, −1 βsin α1 βcos α, the box b=(x,y,w,h,0) becomes straight. The parameter βis needed here to correct different scales along the Xand Y axes—in particular, since W=W 8,H=H 16 ,β=2, we need to stretch the map along the Yaxis twice for proportionality. Again, we can determine the fraction of each cell that falls within the box: wx(i, j)=1−clip x−w 2−¯x ij +0.5+ −clip ¯x ij +0.5−x+w 2, wy(i, j)=1−clip y−h 2−¯y ij +0.5+ −clip ¯y ij +0.5−y+h 2, w(i, j)=wx(i, j)·wy(i, j), i=1,...,H, j=1,...,W. The map of a single feature masked by the box is computed as F t=WFt, vertical aggregation: F t=cos α h[1, ..., 1]×F t (due to the rotation, the intersection of the box with the vertical line has length h cos α, and not just h). C. Straight Boxes with Mask The shape of an object in this case is described not only by a straight box b=(x, y, w, h), but also by a mask P∈ [0,1]H×Wof pixels belonging to the object. Let’s crop this mask to the bounding box using the same wxand wyas in the case of a straight box: P=(wy×[1,···,1]) P, F t=PFt, F t=(F twx)([1, ..., 1] ×P).(1) D. Mask Thinning When constructing ground truth data for boxes, we must keep them high enough so that the cell centers fall within them, and these anchors are assigned to predict boxes. Obviously, the boxes themselves can overlap due to serious line bends and writing density—this will lead to the same features being ”blurred” across different objects, although their text is obviously different. To reduce the damage from this, only at the region pooling stage we can consider all boxes to be one pixel high—this means that in each column information will be read from either one pixel or two adjacent ones. So, •in the case of a straight box his replaced by 1; •in the case of a rotated box his replaced by cos α; •in the case of a straight box with a mask we calculate average vertical level: ˜y=([0.5,1.5,...H−0.5] ×P)([1,1,...,1] ×P) and adjust ˜wy(i)=1−clip((˜y−0.5−i)−clip(i+1−(˜y+0.5)), then recalculating Pand F tin 1 with ˜wyinstead of wy. V. PARAMETERIZATION OF ROTATED RECTANGLES A. Dealing with Parameterization Ambiguity Although different parameterization methods are possible, the conventional bounding box is completely defined by four numbers, for example, the coordinates of its center (x, y), width w, and height h. At first glance, it seems that to define a rotated rectangle, it is enough to add one more parameter to these parameters—the rotation angle α. However, this immediately raises the problem of ambiguity of such a representation, illustrated by Fig. 3: a rotated rectangle can be considered low and wide (height h, width W,hW) with a rotation angle π 4, or high and narrow (height H=W, width h=w) with a rotation angle −π 4. At the same time, the Intersection over Union-based box loss will not suffer, since it is determined by comparing the geometric figures themselves, and not their parameterizations, but the choice of parameterization method can be critical for the Distribution Focal Loss (DFL) [32] if we swap disproportionate height and width. When using DFL, the task of predicting the parameters of the box is set not as a regression problem, but as a soft classification problem. The desired size vis expressed as a weighted sum of the basic sizes {vi=i·d}N i=0: v= N  i=0 wivi,0≤wi≤1, N  i=0 wi=1. The idea of DFL is that among all possible sets of weights {wi}that yield the desired v, the optimal one is considered to be a sparse distribution over the two closest to vbase sizes viand vi+1,i=v d,˜wi=1−v−vi d,˜wi+1 =v−vi d,˜wj=0 if j/∈{i, i +1}. DFL is defined as ldf l (w0,...,w N)=−˜wiln wi−˜wi+1 ln wi+1 and attains a minimum at the optimal set {˜wi}. So the value of DFL depends critically on what is considered width and what is considered height. x y W h w H π 4 −π 4 Fig. 3. Ambiguity of parameterization of a rotated rectangle ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 386 ---------------------------------------------------------------------------- Fixing the range of angles (it is logical to make it symmetrical with respect to 0, which means no rotation, i.e. a straight line) within (−π 4;π 4], although it leads to an unambiguous parameterization, does not remove the problem during training: imagine two low wide rectangles, one with a rotation angle of π 4−ε, the other with an angle of π 4+ε. Then the first of them will receive a parameterization (x, y, w, h, π 4−ε), and the second (x, y, h, w, −π 2+ε), while visually they will be almost indistinguishable. This means that our loss function will be discontinuous in the input, which is the image pixels. A possible solution here is to consider both parameterizations as acceptable, both with a positive and negative angle: ¯ ldf l (b)=min(ldf l (·|w,h),l df l (·|h, w)) . An even finer correction of DFL allows us to pessimize the alternative parameterization at αvalues close to 0: ˆ ldf l (b)=q(α)·ldf l (·|w,h)+ +(1−q(α)) ·min (ldf l (·|w,h),l df l (·|h, w)) . In this case, q(α)should be equal to 1when α=0and equal to 0when α=±π 4. The function q(α)=cos 2(2α)is suitable for this. B. Transition between Parameterizations Recall that in DFL we predict distances from the anchor (¯x, ¯y)to the sides of the rectangle. Let us generalize the necessary formulas to the case of a rotated rectangle specified by the parameters (x, y, w, h, α). When rotating by −α,we move to the case of a straight box with the anchor ¯x ¯y=M¯x ¯y and the center of the box x y=Mx y, where M=cos αsin α −sin αcos α.1 Assuming that d x=x−¯x,d  y=y−¯y, we need to predict the values, shown in Fig. 4: l=w 2−d x,r=w 2+d x,t=h 2−d y,b=w 2+d y. Finally, we will also need the inverse transformation from (t, l, b, r)to to decode the prediction results using DFL: w=l+r, h =t+b, d x=r−l 2,d  y=b−t 2, x y=¯x ¯y+MT·d x d y. 1Note that in the original implementation of YOLOv8 from Ultralytics a serious mistake is made—this rotation is simply not performed, and it is assumed that dx=x−¯x,dy=y−¯y. This is the reason for the very unconvincing quality of the rotated box detectors even in tasks that do not look particularly complex, an example can be seen in figures (a)-(d) on the page https://docs.ultralytics.com/ru/datasets/obb/#yolo-obb-format x y l r b td y d x h 2 w 2 d d (0,0) Fig. 4. Relationship between (t, l, b, r)and (x, y, w, h)parametrizations in an oriented bounding box. The filled dot is the center of the box, the unfilled dot is the anchor. VI. INSTANCE SEGMENTATION BY CENTER PREDICTION A. Segmentation Principle In the classic version of YOLOv8, in the instance segmentation task, not only a bounding box coordinates are predicted for each object, but also a feature description Qof length 32. The same feature description Kmust be obtained for each downsampled pixel of the image. To assess the correspondence of a pixel falling within the box to its object, the value is calculated: v=sigmoid(QTK), this method of assessment resembles the well-known attention mechanism. Note that if a pixel belongs to a unique mask but falls within the boxes of two close objects, their features Q1and Q2must differ drastically from each other so that QT 1Kand QT 2Khave opposite signs. However, it is difficult to single out any visual characteristics that fundamentally distinguish one line of text from another, since they all represent a strip of contrasting thin strokes. For more successful segmentation, we will completely change the principle of assessing pixels to objects and will predict for each pixel (x, y)its offset to the corresponding center of the object, again in the form of soft classification: v= N  i=−N wi·vi,v i=di, N  i=−N wi=1, where vis the offset along xor y. Note that, unlike the distances to the edge of the box considered earlier, these values can be both positive and negative. Also, for each pixel its objectness pis calculated. As a result, we do not calculate keys Kat all, and change the length of Qto 1 + 2(2N+1) while maintaining the rest of the architecture. The pixel is considered to belong to the object with the center (x, y)and the box bif: •it falls within b, •p>t, •there is no object with a closer center in terms of the metric: (¯x+dx−x β)2+(¯y+dy−y)2. Thus, each pixel is tied to no more than one object. The β parameter is needed for more accurate processing of horizontally elongated objects: the deviation along yis considered more critical, and βis set equal to 10. ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 387 ---------------------------------------------------------------------------- The proposed approach is illustrated by Figure 5 and is based on the work of [33]. Fig. 5. Instance segmentation mechanism. Black and white background reflects the visibility pof pixels, lines show the offset vectors (dx,d y), dark dots show predicted box centers for different pixels B. Loss Function Based on the segmentation principle, the loss function should reflect •whether an image pixel belongs to an object (denote the union of all object masks by F) or to the background, for which binary cross-entropy is used: Lfg =−1 WH (¯x,¯y)∈F ln p(¯x, ¯y)+ + (¯x,¯y)/∈F ln(1 −p(¯x, ¯y)), •accuracy of predicting the box center (x, y)using the mean square error: Ldist =1 |F| (¯x,¯y)∈F¯x+dx−x(¯x, ¯y) β2 + +(¯y+dy−y(¯x, ¯y))2, •also for a more concentrated prediction of offsets to the center, the DFL considered earlier is used: Ldf l =1 β2|F| (¯x,¯y)∈F ldf l (w(x) −N(¯x, ¯y),...,w(x) N(¯x, ¯y))+ +1 |F| (¯x,¯y)∈F ldf l (w(y) −N(¯x, ¯y),...,w(y) N(¯x, ¯y)). VII. FORMATTING RECOGNITION Let us assume that we are recognizing not only a symbol from the alphabet A, but also its style from the set S. In this case, the symbol and the style are recognized independently, i.e. p(a, s)=p(a)p(s),a∈A,s∈S. For this, the output of the last layer Vbefore feeding into the CTC loss has the dimension |A|+|S|+1, the first |A|+1 values is responsible for the symbol, including the empty symbol ε, the last |S|— for its style. We transform these values to get the probability of the combination: wi=logsoftmax(vi|v1,...,v |A|+1), i=1,...,|A|+1, uj=logsoftmax(vj|v|A|+2,...,v |A|+|S|+1), j=|A|+2,...,|A|+|S|+1, pt=wmod(t−1,|A|)+1 +udiv(t−1,|A|)+1, t=1,...,|A|·|S|, p|A|·|S|+1 =w|A|+1, that is, CTC loss is calculated in the alphabet of |A|·|S|+1 “symbols-styles” using the matrix P, an empty symbol is not divided into styles. This approach allows predicting with CTC loss even those symbols that were not present in the required style in the training sample, since the symbol and style are predicted separately. VIII. EXPERIMENTS A. Data preparation The initial data are the diaries of Modest Andreevich Korf (1800–1876), one of the outstanding representatives of Russian conservative thought in the mid-19th century, a member of the State Council, senator, and actual privy councilor. Korf’s diary archives are extremely extensive and contain about 9,000 pages. Of particular interest is the excellent systematization of the diary: each volume is preceded by an alphabetical index listing the entities mentioned—persons, places, events— with corresponding page numbers. The bulk of the entries are in a single format—the text is given in a narrow column on the right half of the page, the left is used for notes. Text transcriptions for the 100 pages of the third volume, dating back to 1840, were restored from the edition [34]. The text was broken into lines, a total of 4532, and checked for assignment to bounding boxes by an expert. Also, for a more stable allocation of boxes in the training, 40 pages with an alphabetical index of 1305 lines without a text transcript from the 1st, 2nd and 3rd volumes were used. The test and validation samples included 20 images each from different volumes. To avoid fragments of lines from other pages in the frame that do not have markup, these fragments were colored in the background color, for which segmentation into an object and background was carried out using Kraken OCR [1]. An example of such a correction is shown in Figure 6. To describe the shape of the lines, the same Kraken OCR was used to extract the baselines of the text with subsequent ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 388 ---------------------------------------------------------------------------- (a) (b) (c) Fig. 6. Removing unnecessary text fragments. (a) Original page, (b) mask of redundant fragments, (c) corrected page. manual correction. Let the baseline bbe given as a polyline {(xi,y i)}n i=1. Let us define the corresponding point of the upper and lower envelope for each vertex: x(t) i=xi−dtcos αi,y (t) i=yi−dtsin αi, x(b) i=xi+dbcos αi,y (b) i=yi+dbsin αi, where αiare calculated as bisectors of two links of the polyline adjacent to the point (xi,y i)or as perpendiculars of the extreme links for the ends of the polyline. The offsets dtand dbare equal to 9 1000 and 7 1000 from the image height, respectively. A mask of arbitrary shape is defined by a polygon obtained by going over the points (x(t) 1,y(t) 1),...,(x(t) n,y(t) n),(x(b) n,y(b) n),...,(x(b) 1,y(b) 1). The rotated box is constructed as a minimal rectangle covering all these points, and the straight box is constructed by the minimal and maximal xand y. All types of defining the shape of lines are demonstrated in Figure 7. B. Training process The need to develop specialized networks adapted to specific handwriting is caused by the lack of ready-made models that provide sufficient recognition quality. In particular, the Russian generic handwriting 2 model2, the most relevant for recognizing Russian text among Transkribus models, showed results of 31.10% CER and 70.53% WER, on our test sample. The TrOCR-ru transformer model3produces more than 50% errors in characters while operating extremely slowly. Large language models like ChatGPT 5.0 and Gemini 2.5 Pro require at least thorough prompt engineering combined with few-shot learning, otherwise, with a basic prompt, they produce only partially relevant, albeit readable, text. To reduce the time costs and demonstrate the possibilities of transfer learning, the YOLO-HTR model with straight boxes, previously trained for recognizing the handwriting of Admiral F.P. Litke [31], was considered as a starting point. In places where the model architecture changed, new layers were left initialized randomly. In the last layer of the model, which calculates the probabilities of symbols, the weights of 2https://app.transkribus.org/models/public/text/ russian-generic-handwriting-2 3https://huggingface.co/kazars24/trocr-base-handwritten-ru (a) (b) (c) (d) Fig. 7. Ground truth line segmentation by baselines (a) for (b) detection, (c) OBB detection, (d) segmentation problems the symbols that Litke had were copied from the original model, the weights of the symbols found only in Korf were left random, as were the weights responsible for the style— recognition of the style in Litke was not considered. The values of the weights of the loss function termns were set as Lbox =7.5,Lcls =0.5,Ldf l =1.5,Lctc =0.1, Lseg =7.5. Due to the large size of the images, equal to 2048, they were presented individually during training, without combining them into batches. For images without ground truth text, the CTC loss was not calculated. All other parameters were taken from the YOLOv8 implementation by Ultralytics [35], including the Adam optimizer with a learning rate of 0.002 and a momentum of 0.9. The training process, visualized in Figure 8, shows the ability to successfully search for lines in any format on an ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 389 ---------------------------------------------------------------------------- image. At the same time, searching for rotated boxes turns out to be slightly more difficult than searching for straight ones, which is evident from the most complex metric mAP5095 (mean average precision, averaged over IoU in the range from 0.5 to 0.95)—apparently due to sensitivity to errors in restoring the rotation angle. 0 50 100 150 200 250 300 350 40 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 val/box_loss train/box_loss val/cls_loss train/cls_loss val/dfl_loss train/dfl_loss val/ctc_loss train/ctc_loss 0 50 100 150 200 250 300 350 40 0 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 metrics/precision(B) metrics/recall(B) metrics/mAP50(B) metrics/mAP50-95(B) 1-val/cer 1-val/wer (a) (b) 0 100 200 300 400 500 600 700 800 900 1000 0 0.5 1 1.5 2 2.5 val/box_loss train/box_loss val/cls_loss train/cls_loss val/dfl_loss train/dfl_loss val/ctc_loss train/ctc_loss 0 100 200 300 400 500 600 700 800 900 1000 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 metrics/precision(B) metrics/recall(B) metrics/mAP50(B) metrics/mAP50-95(B) 1-val/cer 1-val/wer (c) (d) 0 50 100 150 200 250 300 350 400 45 0 0 1 2 3 4 5 6 7 8 9 val/box_loss train/box_loss val/cls_loss train/cls_loss val/dfl_loss train/dfl_loss val/ctc_loss train/ctc_loss val/seg_loss train/seg_loss 0 50 100 150 200 250 300 350 400 45 0 0.7 0.75 0.8 0.85 0.9 0.95 1 metrics/precision(B) metrics/recall(B) metrics/mAP50(B) metrics/mAP50-95(B) 1-val/cer 1-val/wer metrics/precision(M) metrics/recall(M) metrics/mAP50(M) metrics/mAP50-95(M) (e) (f) Fig. 8. Values of the loss function (a,c,e) and quality metrics (b,d,f) for (a-b) Detect, (c-d) OBB, (e-f) Segment models. Metrics related to boxes are marked (B), and to masks (M). Only Segment model has metrics related to masks. It is noteworthy that since the task of predicting the mask is not posed at all in the original model, training actually begins from scratch and the loss function starts with a high value, as Figure 8e shows. The performance of the models was measured for two configurations, with the results presented in Table II: •Server: Intel Xeon Platinum 8255C / 196GB RAM / NVIDIA Tesla A100 PCIe 40GB •Laptop: Intel Core i7-9750H / 16GB RAM / NVidia GeForce GTX 1660 Ti 6GB The laptop does not have enough video memory for training. Server-based training allows for 600–900 training iterations per day, depending on the model type. C. Effectiveness of Architectural Solutions We will separately study the impact of box and mask prediction accuracy on the quality of text recognition. To do this, we will compare the character and word error rates (CER TABLE II. TIME CONSUMPTION IN SECONDS PER IMAGE. Stage + Hardware Detect OBB Segment Training (Server) 0.887 0.918 1.326 Inference (Server) 0.292 0.304 0.340 Inference (Laptop) 1.262 1.296 1.443 and WER) for ideal boxes/masks and for actually predicted ones. In Table III, the indicators for ideal and actual boxes, when only lines with IoU >0.5 with the ground truth data are taken into account, do not differ significantly. The slight advantage of actual boxes over ground truth ones can be explained by the fact that when calculating CER and WER for the former, missed difficult boxes are simply ignored, and they, as a rule, contain more challenging text, for example, interline insertions. Also, the original line-level VAN model, which is essentially YOLO-HTR+ without a box decoder, was trained on a sample of lines obtained by straightening their polygons relative to the baseline using piecewise perspective transformation (Fig. 9). The VAN was unable to outperform the segmentation model in quality, which speaks, on the one hand, to the importance of straightening lines (YOLO-HTR+Segment does this in the feature space), and on the other hand, to the benefit of taking into account the context that is lost when switching to linelevel sampling. Also note that when considering VAN, the line detection problem itself is ignored, and its solution is considered ideal. TABLE III. THE IMPACT OF LINE DETECTION ACCURACY ON THE QUALITY OF TEXT RECOGNITION. THE INDICATORS, AS BELOW, ARE GIVEN IN %. Model CER WER predicted ground truth predicted ground truth Detect 3.846 4.060 14.17 14.69 OBB 3.533 3.713 13.45 13.93 Segment 3.227 3.281 12.64 12.66 Line-level (VAN) −3.440 −13.25 Fig. 9. Samples from the line-level dataset We will also evaluate the usefulness of the move with reducing the height of the lines to 1 pixel, comparing the results with and without this normalization. The results in Table IV indicate the effectiveness of this technique, especially for straight boxes, which is explained by the strong overlap of the regions of interest without normalization. For rotated lines, the effect is least significant, presumably because the shape of the object itself changes the least with normalization. ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 390 ---------------------------------------------------------------------------- TABLE IV THE EFFECT OF BOX THINNING ON TEXT RECOGNITION QUALITY Model CER WER norm. non-norm. norm. non-norm. Detect 3.846 10.46 14.17 32.83 OBB 3.533 3.848 13.45 14.92 Segment 3.227 3.921 12.64 14.87 The justification for developing a new mechanism for instance segmentation is demonstrated by Figure 10 with an example of processing an image with strongly inclined lines. As can be seen from the sequence of degraded, broken masks at the bottom of the left image, the original YOLOv8 mechanism insufficiently distributes pixels along lines with intersecting frames. This is confirmed by the numerical estimates from Table V, where a deviation from the near-perfect segmentation of our method results in an increase in CER of nearly 2.5 percentage points. (a) (b) Fig. 10. The result of the instance segmentation model: (a) Initial, based on the key-query mechanism, (b) proposed, based on the prediction of the box center. TABLE V COMPARISON OF INSTANCE SEGMENTATION METHODS.SEGMENTATION QUALITY METRICS ARE GIVEN FOR MASKS. Model CER WER Precision Recall mAP50 mAP50-95 Original 5.680 17.27 97.54 97.46 97.23 73.13 Proposed 3.227 12.64 99.38 99.15 99.42 88.19 Text formatting in Korf’s diaries includes four options: regular text, strikethrough, underline, and superscript. Since ignoring formatting simplifies the task, the percentage of text recognition errors decreases, as can be seen from Table VI. Note that since the share of non-trivially formatted text is 2.22%, and the difference in CER is 1.0-1.3%, we can estimate the share of formatting omissions at 8-10%. Finally, we check which images show the difference in text recognition quality by analyzing the data from Table VII (only 14 out of 20 images in the test sample have reference text). TABLE VI TEXT RECOGNITION QUALITY WITH AND WITHOUT FORMATTING Model CER WER char-style char only char-style char only Detect 3.846 3.717 14.17 13.90 OBB 3.533 3.394 13.45 13.21 Segment 3.227 3.125 12.64 12.50 TABLE VII RECOGNITION QUALITY FOR INDIVIDUAL IMAGES (CER IN %) #Image Detect OBB Segment 1 3 12rev. 3.127 2.971 2.971 2 3 16 3.465 3.612 3.371 3 3 19rev. 7.316 5.918 5.270 4 3 61 3.918 3.437 3.184 5 3 30 2.763 2.016 2.390 6 3 105rev. 3.156 3.024 3.156 7 3 26rev. 5.651 4.751 4.083 8 3 68 3.152 2.189 1.751 9 3 71rev. 4.134 4.690 4.372 10 3 64rev. 4.009 3.480 3.933 11 3 75 3.655 3.421 3.188 12 3 33rev. 2.524 2.004 1.856 13 3 23 2.986 2.403 2.403 14 3 37 3.987 3.255 3.255 It turns out that the greatest increase in recognition quality when moving to a more complex model is achieved with a fairly complex format of the original lines, namely: •the presence of curvature, and not just a slope; •the presence of abundant interline insertions; •the presence of fragments with a tight arrangement of lines. An example of such a page is 3 19rev., shown in Figure 11. It can be noted that the segmentation model recognizes the formatting more successfully. IX. CONCLUSION The paper explored the principles of constructing neural network architectures capable of solving the problems of handwritten text line search and recognition within a common model. The YOLOv8 was chosen as a model, whose backbone was replaced with layers better suited for extracting handwritten character features. Since these layers have a fairly narrow receptive field, the basic architecture was modified so that the cells responsible for line detection could see the lines in full width with a small number of additional parameters. The architectures for three different problems—object detection, oriented bounding box detection, and instance segmentation—were modified to match three line shape options of increasing complexity—straight, rotated, and curved lines. At the same time, theoretical problems associated with the ambiguity of coding rotated rectangles were discovered and solved. Based on the features of the detected objects—an elongated shape and visual similarity—a new method of instance segmentation was proposed, based on a purely geometric principle. The performance of the designed models was analyzed ISSN 2305-7254________________________________________PROCEEDING OF THE 38TH CONFERENCE OF FRUCT ASSOCIATION ---------------------------------------------------------------------------- 391 ----------------------------------------------------------------------------