Go to originalLocalized version was produced by AI. It may contain inaccuracies.
Preparing document pages…
Page 1
VIRGINIJUS DADURKEVIČIUS
Vilnius University
LITHUANIAN LANGUAGE MORPHOLOGY ON THE OPEN-SOURCE HUNSPELL PLATFORM1
KEYWORDS: computational linguistics, Lithuanian, grammar, morphology, morphological analysis, spell-checking, text indexing, search, open source, Hunspell.
INTRODUCTION
More than 100 years ago, through the efforts of Jonas Jablonskis (1901, 1918, 1922), the Lithuanian language began to be standardized and adapted to serve as the principal means of uniting the state. Already in Jonas Jablonskis’s first Lithuanian Grammar (1901), Lithuanian phonetics, morphology, and syntax were formalized: basic concepts were defined, and rules for word inflection and sentence structure were presented (see Fig. 1).
FIG. 1. Excerpt from Jonas Jablonskis’s Lithuanian Grammar 19012
Jonas Jablonskis made an invaluable contribution to standardizing the Lithuanian lexicon and explaining the semantic characteristics of words.
For a long time, this was entirely sufficient for students and teachers, writers and readers, civil servants and scholars. With the advent of the civilization-changing innovation of computers, classical grammar and lexicon faced the need to “put on a new garment” and become full-fledged in the new
1 This article is based on the paper Lithuanian Grammar in the Open-Source World, presented at the 24th International Jonas Jablonskis Conference, Digital Language Resources: Development Directions and Uses (2017 09 29); it was organized by the Centre for Standard Language Research of the Institute of the Lithuanian Language and the Department of Lithuanian Language of Vilnius University.
2 Figure from http://www.epaveldas.lt/vbspi/biRecord.do?biRecordId=25383.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351–7204
Page 2
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 2
part of technology. First, language-related tools were needed in computers for encoding text (converting letters into numbers), automatically correcting errors in entered text, and hyphenating text in columns with bilateral justification. Somewhat later, they were needed for information retrieval, speech analysis and synthesis, machine translation from one language to another, automatic content comprehension, “sentiment” analysis of text, and so on. Language-related tools had to be meticulously formalized and recorded in another way that computers could understand. The most important of these is morphology, since it is the foundation for any more in-depth computational linguistics work (syntax, semantic analysis, and so on).
The first such work was carried out at the turn of the century by Vytautas Zinkevičius (1996, 2000), a programmer at the experimental factory “Bitas.” For almost two decades, it was an indispensable and sole tool for modern Lithuanian language research and computer applications. The system was successfully used to detect spelling errors in word-processing programs and to perform morphological analysis and synthesis in research and applied projects at the Institute of the Lithuanian Language, Vytautas Magnus University (where his work was named “Lemuoklis”), and Vilnius University. A version of the system adapted to analyze Old Lithuanian was even developed (Gelumbeckaitė et al. 2012). However, as the needs of computational linguistics expanded, the system’s shortcomings began to emerge: non-standard, undocumented data structures; closed source code; and incomplete implementation of proper names, the illative, the dual, the future tense of primary verbs, shortened forms, and rare forms. Closed source code especially impeded the system’s further development: in practice, only the author himself could modify the data and algorithm.
To avoid the shortcomings described above and create a new-generation computational morphology for Lithuanian, the following requirements were established:
1) use and develop only open-source code;
2) only the data, not its form or interpretation, may have specific Lithuanian-language features;
3) all program code must be universal and suitable for any other language;
4) make use of solutions successfully adapted to other languages of the world.
The Hunspell platform3 meets the aforementioned requirements well. Moreover, once a language has been described in this way, spelling errors can later be checked easily in many (over 50) applications (OpenOffice, LibreOffice, Firefox, Chrome, Safari, InDesign, etc.), words can be morphologically analyzed and synthesized, intelligent searches of textual information can be carried out, and so on.
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 3
This article reviews a new implementation of Lithuanian morphology based on the Hunspell platform. It discusses related problems and ways of addressing them, and presents examples of successful testing and practical application.
1. FEATURES OF THE HUNSPELL FORMAT FOR DESCRIBING LITHUANIAN
1.1. Prerequisites for the emergence of the Hunspell platform and general principles
The open-source Hunspell platform evolved from the simplest spell-checking tools, Ispell4, Aspell5, MySpell6, and others. As these tools gradually became more sophisticated and were adapted to an increasingly diverse range of languages (especially agglutinative languages such as Finnish, Hungarian, Turkish, and so on), it became possible to address not only spell-checking tasks but also morphological analysis and synthesis. The creators of this extremely popular platform worldwide are mainly scientists and programmer-engineers of Hungarian origin (Trón et al. 2005). It is no surprise that the first part of its name, “Hun-,” comes from the English name for Hungary.
A language’s morphology is described in the Hunspell platform using two text files: inflection rules are recorded in a file with the .aff extension, while a word-form dictionary containing basic morphological information and references to one or more groups of inflection rules that may additionally be applied is recorded in a file with the .dic extension. An individual rule can be thought of as an instruction specifying:
1) What can be changed into what in a word from the word-form dictionary. For example, in the rule
SFX 85 čias ty [^š]čias is:Masc_Sg_Voc
“SFX” indicates that the end of a word is changed, and that “-čias” can be changed to “-ty.”
“PFX” would indicate that the beginning of a word is changed. Changes to the middle of a word are not provided for.
2) The number of the inflection-rule group to which this rule is assigned. A group may consist of one or more rules that are always applied together. In the preceding example, the rule is assigned to the group numbered “85.”
3) The conditions under which the change is possible (a certain formalism of regular expressions is used). In our example, the change is possible only if the word from the word-form dictionary does not end in “-ščias” (e.g., “vaikiščias”).
4) The morphological information associated with this change. In our example, this change to the end of a word is associated with the morphological tag “Masc_Sg_Voc” (masculine, singular, vocative, e.g., “svety”).
4 Available online: https://www.cs.hmc.edu/~geoff/ispell.html.
5 Available online: http://aspell.net.
6 Available online: https://code.google.com/archive/a/apache-extras.org/p/ooo-myspell.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351–7204
Page 4
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 4
The principles of composing Hunspell inflection rules and specifying morphological information can be explored in more detail on the platform’s website7, while the possibilities for describing the morphology of various languages on this platform are examined quite thoroughly in Tommi A. Pirinen’s dissertation (2014).
1.2. Specific requirements for the Lithuanian implementation
For such a formalization of language to be suitable for morphological analysis and synthesis, and for indexing and searching textual information, the following two requirements must be met:
1) only lemmas (citation forms) may be entered in the word-form dictionary;
2) prefixal, suffixal, and reflexive derivation must be represented not in the .aff file, but in the .dic file.
The first requirement is not easy to implement for Lithuanian because of the great diversity of verb inflection (about 170 variants), but the problem was solved with the help of the Hunspell platform’s ability to call one inflection rule from another. Although the “depth” of such calls can be only one, this considerably reduces the size of the inflection-rules file—from several million lines to tens of thousands.
Meeting the second requirement somewhat increases the size of the .dic file because, for example, prefixed, suffixed, and reflexive derivatives of the same root—“imti,” “paimti,” “suimti,” and “nesiimti”—will be entered on separate lines. And this is not the only disadvantage of this requirement: it is easy to make a mistake and leave out some less frequently used derived verb or adjective. This problem, however, is addressed quite thoroughly by selecting lemmas primarily from collected corpora rather than from published dictionaries.
1.3. Fragments (examples) of Lithuanian.dic and.aff files
Several fragments of a Lithuanian .dic file (dictionary), with the equivalents of Jono Jablonskio’s exemplary words highlighted, are shown in Figure 2.
7 Available online: https://github.com/hunspell/hunspell.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204
Page 5
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 5
FIG. 2. Fragment of the .dic file containing word forms. The numbers after the slash indicate references to the addressed inflection-rule groups. References to the rule groups shown in FIG. 3 are highlighted in red.
Fig. 3 shows several fragments of the .aff file containing Lithuanian inflection rules:
SFX 21 Y 1
SFX 21 as uosna. is:Masc_Pl_I
...
SFX 98 Y 2
SFX 98 as ai. is:Masc_Pl_Nom
SFX 98 as ų. is:Masc_Pl_Gen
...
SFX 264 Y 6
SFX 264 is is. is:Masc_Sg_Nom
SFX 264 ys ys. is:Masc_Sg_Nom
SFX 264 is žio. is:Masc_Sg_Gen
SFX 264 ys žio. is:Masc_Sg_Gen
SFX 264 ai ai. is:Masc_Pl_Nom
SFX 264 ai ų. is:Masc_Pl_Gen
...
SFX 145 Y 12
SFX 145 is iams. is:Masc_Pl_Dat
SFX 145 ys iams. is:Masc_Pl_Dat
SFX 145 is iam. is:Masc_Pl_Dat_short
SFX 145 ys iam. is:Masc_Pl_Dat_short
SFX 145 is ius. is:Masc_Pl_Acc
SFX 145 ys ius. is:Masc_Pl_Acc
SFX 145 is iais. is:Masc_Pl_Inst
SFX 145 ys iais. is:Masc_Pl_Inst
SFX 145 is iuose. is:Masc_Pl_Loc
SFX 145 ys iuose. is:Masc_Pl_Loc
SFX 145 is iuos. is:Masc_Pl_Loc_short
SFX 145 ys iuos. is:Masc_Pl_Loc_short
FIG. 3. Fragments of the inflection rules .aff file. Several rules highlighted in bold are used below to illustrate morphological analysis.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351–7204
Page 6
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 6
Using the language description provided in the examples and the standard (language-independent) Hunspell analysis tools, we would obtain the following response for the example words from Jonas Jablonskis (see Fig. 4).
FIG. 4. Example of morphological word analysis
2. WORKFLOW AND RESULTS
Once a complete language description in Hunspell format has been compiled (i.e., the corresponding .dic and.aff files have been created), practically unlimited possibilities for applications become available—from spell- or grammar-checking to intelligent information retrieval systems. Because of the fairly complex morphology of Lithuanian (many rules and rule groups) and the need to reconcile the lemma list with large corpora (more than 1 billion words) reflecting the actual state of contemporary written Lithuanian, it took approximately five years from the first attempts to develop a smoothly functioning morphology. The examples presented earlier in this article are more illustrative, intended only to help understand the idea; in reality, the rules and their groups are much more complex, especially those related to verbs.
To speed up the development of the morphology and avoid errors when writing thousands of rules and assigning them to lemmas, an initial, more abstract description of the morphology was first created, beyond that required by the Hunspell specifications (rules—in an MS Excel spreadsheet; lemmas and their classification—in an MS Access database). Software tools were also developed to automatically transform these initial structures, which are easier for people to read and understand, into .aff and.dic files or another format specified by the application. The latest version of the Lithuanian .aff and.dic files is available on the author's GitHub website8.
The following main sources were used throughout the process of formalizing the morphology:
8 Available online: https://github.com/dadurka/hunspell_morphology_lt.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351–7204
Page 7
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 7
1) Grammar of Contemporary Lithuanian (DLKG 2006);
2) The Corpus of Contemporary Lithuanian of Vytautas Magnus University (hereinafter VDU)9 – ~140 million words;
3) The machine translation corpus of Vilnius University10 – ~800 million words, ~4 million unique items;
4) Documents of the Parliament of the Republic of Lithuania (hereinafter LRS)11 – ~400 million words, ~1 million unique items;
6) Dictionary of the Lithuanian Language12 – ~500 thousand entries;
7) Dictionary of International Words (VTŽŽe) – ~22 thousand lemmas;
8) Dictionary of Lithuanian Surnames (LPŽ) (~80 thousand surnames).
In total, approximately 5,000 addressable rule groups (18,000 individual rules) were created. The distribution of rule groups by part of speech is shown in Figure 5.
FIG. 5. Distribution of inflectional rule groups by part of speech
.dic file (dictionary) was compiled from 171,000 lemmas. The distribution of lemmas by part of speech is shown in Figure 6.
9 Available online: http://tekstynas.vdu.lt/tekstynas/.
10 Available online: https://www.versti.eu/.
11 Available online: http://www.lrs.lt/.
12 Available online: http://www.lkz.lt/.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204
Page 8
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 8
FIG. 6. Distribution of lemmas by part of speech
During selection, only correct contemporary Lithuanian was considered, and incorrect, non-Lithuanian, nonstandard, offensive, and obsolete words were avoided. Proper nouns were further divided into surnames, given names, geographical names, and other cases. This selection facilitates the application of the developed morphology to spell-checking and makes it easier to apply the analysis results in subsequent stages of text analysis (syntax, named-entity recognition, indexing, and search).
The accuracy of the newly developed morphological analyzer is about 98 percent; that is, on average, the analyzer correctly interprets 98 out of every 100 words (Kapočiūtė-Dzikienė et al. 2017). The remaining 2 percent of unrecognized words most often consist of spelling errors, words from other languages (given names, surnames, phrase quotations, etc.), nonstandard or non-normative words, and correct words that were not included in the lemma list.
3. PRACTICAL APPLICATION
3.1. VDU Syntactic-Semantic Analysis System
The newly developed Lithuanian morphology on the Hunspell platform was adapted for use in the VDU Syntactic-Semantic Analysis System13 from 2011 to 2015 for spelling and grammar checking and for morphological and syntactic text analysis (see Fig. 7). Morphological analysis was supplemented with hybrid statistical-rule-based disambiguation (a hidden Markov model, trigram probabilities approximated according to the gold standard, the Viterbi algorithm, and several exceptions). A more detailed discussion of disambiguation in morphological analysis appears in the publication by Jurgita Kapočiūtė-Dzikienė, Erika Rimkutė, and Loïc Boizou (2017).
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 9
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 9
FIG. 7. Hunspell application of Lithuanian morphology in the VDU Syntactic-Semantic Analysis System
3.2. Search system of the LRS Register of Legal Acts
In 2011–2013, alongside the development of a new Lithuanian morphology, it was adapted for use in the Register of Legal Acts search system at the LRS Chancellery14. Usually, in such systems, before the first search is performed, all documents are indexed by lemmatization, i.e., all words in the documents are converted to their lemmas, and it is recorded where each lemma occurs. Before a search query is run, its text is also lemmatized in the same way, so that not only the word form written in the query is found, but also any other form of that word occurring in the documents being searched. Since it was not possible to use the .dic,.aff data directly in this system, the original format required by the system (a text file in which each line is [lemma] [part of speech] → [list of forms]) was generated from the primary morphology description mentioned earlier. Although this description can generate about 15 million theoretically possible word forms, due to internal limitations of the search system only about 1 million of the most frequent forms were used. This solution proved successful: the search works quite smoothly and is still in use (see Fig. 8).
14 Available online: https://www.e-tar.lt
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351–7204
Page 10
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 10
FIG. 8. Application of Lithuanian morphology in the LRS Register of Legal Acts search system
4.3. Open-source document indexing and search system Solr/Lucene15
To make broader use of the advantages of open source and the experience gained in implementing Lithuanian morphology in the Register of Legal Acts, a system for indexing and searching Lithuanian documents was developed on the basis of Solr/Lucene. Direct use of the .dic,.aff data is possible in this system, but very inefficient, so it was replaced with a finite-state transducer (FST – Finite State Transducer), created from the primary morphology description mentioned earlier and generating all word forms theoretically possible according to that description. This solution makes it possible not only to index and find all desired word forms accurately, but also to speed up indexing hundreds of times, while obtaining search results by using various semantic relations and taking additional taxonomic-semantic information into account. For example, if the query specifies the word “snake,” the system also searches for “viper,” “rattlesnake,” and so on (see Fig. 9), whereas if the query specifies “viper,” the search results will include neither “snakes” nor “rattlesnakes” (Ralys, Dadurkevičius 2017).
15 Available online: http://lucene.apache.org/solr/.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204
Page 11
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 11
FIG. 9. Example of applying Lithuanian morphology in the open-source document indexing and search system Solr/Lucene
4. PERSPECTIVES
To apply the implementation of Lithuanian morphology on the Hunspell platform more broadly, two directions should be developed: a standard-language version and a full version (standard and non-standard language). The first version would continue to support spelling-checking tasks and provide recommendations for correcting violations of language norms, while the second would be suitable for applied tasks involving the analysis of living language.
An equally important task for the future is to continually supplement the morphology dictionary with omitted and newly emerging words, correct identified errors in inflection rules and word classification, and publish the latest .dic,.aff versions at least once a year.
Drawing on the experience already accumulated, it would also be easy to prepare a formal description of Lithuanian dialects (especially Samogitian) or of the language of old Lithuanian writings.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204
Page 12
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 12
ŠALTINIAI
DLKG 2006 – Dabartinės lietuvių kalbos gramatika. Red. Vytautas Ambrazas. 4-oji pataisyta laida, Vilnius: Mokslo ir enciklopedijų leidybos institutas.
DŽ 2006 – Dabartinės lietuvių kalbos žodynas (elektroninis). Vyr. red. Stasys Keinys. 6-as leid. Vilnius: Lietuvių kalbos institutas.
LKŽe – Lietuvių kalbos žodynas, elektroninis variantas. Red. kolegija: Gertrūda Naktinienė (vyr. red.), Jonas Paulauskas, Ritutė Petrokienė, Vytautas Vitkauskas, Jolanta Zabarskaitė, Vilnius: Lietuvių kalbos institutas, 2005. Atnaujinta versija, 2008. Prieiga internete: www.lkz.lt.
LPŽ – Aleksandras Vanagas (red.), Lietuvių pavardžių žodynas 1–2, Vilnius: Mokslas, 1985–1989.
VTŽŽe 2003 – Vaitkevičiūtė V. Tarptautinių žodžių žodynas (elektroninis). Vilnius: Fotonija.
LITERATŪRA
Gelumbeckaite ir kt. 2012: Gelumbeckaite J., Šinkūnas M., Zinkevičius V. Old Lithuanian Reference Corpus (SLIEKKAS) and Automated Grammatical Annotation. – Journal for Language Technology and Computational Linguistics (JLCL), 2012. - 27.2. Altüberlieferte Sprachen als Gegenstand der Texttechnologie. Ancient Languages as the Object of Text Technology, ed. by Armin Hoenen, Thomas Jügel, Gesellschaft für Sprachentechnologie und Computerlinguistik (GSCL), 83–96 p.
Jablonskis J. 1901: Lietuviškos kalbos gramatika. Rašytojams ir skaitytojams vadovėlis. Parašė Petras Kriaušaitis. Tilžėje 1901. Spausdinta pas Otto v. Mauderode. 88 p.
Jablonskis J. 1918: Lietuvių kalbos gramatika: etimologija: pirmosioms mūsų aukštesniosioms mokslo įstaigoms / P. Kriaušaičio ir Rygiškių Jono. Vilnius: Lietuvių mokslo draugija. 237 p.
Jablonskis J. 1922: Lietuvių kalbos gramatika: etimologija: vidurinėms mokslo įstaigoms / Rygiškių Jonas. 2-asis leid. Kaunas; Vilnius: „Švyturio“ b-vė, (Tilžė: Otto v. Mauderodės sp.). 280 p.
Kapočiūtė-Dzikienė J. ir kt. 2017: Jurgita Kapočiūtė-Dzikienė, Erika Rimkutė, Loïc Boizou Comparison of Lithuanian Morphological Analyzers. – Text, Speech, and Dialogue. 20th International Conference, TSD 2017, Prague, August 27–31, 2017, Proceedings, 47–56.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204
Page 13
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 13
Ralys D., Dadurkevičius V. 2017: Informacijos paieškos sistemos ir lietuvių kalba, [žiūrėta 2017-10-30]. Prieiga internete:
Tommi A Pirinen 2014: Weighted Finite-State Methods in Spell-Checking and Correction, Doctoral dissertation, University of Helsinki.
Trón V. ir kt. 2005: Viktor Trón, András Kornai, György Gyepesi, László Németh, Péter Halácsy, and Dániel Varga. 2005. Hunmorph: open source word analysis. – Proceedings of the Workshop on Software (Software '05). Association for Computational Linguistics, Stroudsburg, PA, USA, 77–85.
Zinkevičius V. 1996: Lietuvių kalbos morfologinių reiškinių kompiuterizavimo lingvistiniai aspektai. – Lietuvių katalikų mokslų akademijos suvažiavimo darbai, XVI tomas, 148–154.
Zinkevičius V. 1996: Lietuvių kalbos morfologinių reiškinių kompiuterizavimas. – Lietuvių katalikų mokslų akademijos suvažiavimo darbai, XVI tomas, 155–162.
Zinkevičius V. 2000: Lemuoblis – morfologinei analizei. – Darbai ir dienos 24, 245–273.
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 14
LITHUANIAN MORPHOLOGY IN THE HUNSPELL FRAMEWORK
Summary
The paper presents the results of an attempt to build basic Lithuanian language resources using the widespread Hunspell platform. The spelling is actually the primary target of this open-source platform but the morphological analysis and synthesis are also possible. Moreover, the ability to efficiently perform lemmatization (stemming) makes this platform the best option for text search engines (e.g. Solr/Lucene) and information retrieval. Taggers, grammar checkers and other basic natural language processing tools can also be build using properly built Hunspell language resources.
Every Hunspell language resource consists of two files: dictionary and affixes (it may be empty). The dictionary contains main forms (lemmas) whereas the affixes contain the morphological rules to generate all possible forms. As a source for the dictionary we have used the Modern Lithuanian Dictionary (6-th edition), Corpus of the Contemporary Lithuanian Language compiled at the Center of Computation Linguistics of Vytautas Magnus University, database of documents of the Lithuanian Parliament, versti.eu machine translation corpus of Vilnius University and various public internet sources (totally 1.3 billion tokens). Main criteria for semi-manual compilation of the Lithuanian dictionary of lemmas from these sources was correctness, usability, actuality and approval by language authorities. Deprecated loanwords or extremely rare, exotic, obsolete, jargon, insulting forms were discarded from the list. Resulting dictionary consists of 171 000 lemmas: 42 000 common nouns, 73 000 proper nouns, 15 000 adjectives, 53 pronouns, 153 numerals, 35 000 verbs, 4 000 adverbs and 2 000 others (prepositions, conjunctions, particles, onomatopoeias, interjections, acronyms and abbreviations).
The second component of language resource, the so called “affix file”, contains information of various kind: metadata, preferable suggestions for spelling correction, grouping of rules, explicit tags for flexing and non-flexing properties, rules for suffix and affix alteration.
In order to make the Hunspell resources suitable for creating basic language tools, e.g. morphological analyzer and synthesizer, some principles should be kept:
1) every flexion paradigm (consisting of one or more rules) should be thoroughly generated from one single lemma in dictionary file (it is not trivial, especially for irregular verbs);
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351–7204
Page 15
VIRGINIJUS DADURKEVIČIUS. Lietuvių kalbos morfologija atvirojo kodo hunspell platformoje | 15
2) every individual alteration case should have its own morphological tag, e.g. ‘Masc_Sg_Il’ for masculine + singular + illative;
3) every dictionary item should have references for part of speech and other non-flexing information;
4) avoid prefixation via rules, use dictionary instead – affixed forms may have completely different meanings and using them under single lemma may cause problems for text search engines;
5) do not rely much on calling rules from rules – calling depth can by no more than 1.
The coverage of the contemporary Lithuanian by this implementation of Lithuanian morphology is about 98 percent. The full list of all the theoretically possible forms generated by this resource contains about 17 million entries.
This work clearly shows an efficient way for any language (especially with scarce funding resources) to make basic language tools using a single open source development platform – the Hunspell.
