Lietuviško balso sintezės dabartis ir perspektyvos

PIJUS KASPARAITIS, GINTARAS SKERSYS Vilniaus universitetas THE PRESENT AND FUTURE OF LITHUANIAN SPEECH SYNTHESIS KEYWORDS: Lithuanian language, speech synthesis, LIEPA project. INTRODUCTION Speech is the most natural and acceptable means of communication between humans and computers, so it is no surprise that considerable effort is devoted to improving computers’ ability to speak in various languages. Lithuanian is no exception. Although a quarter of a century has passed since the first attempts to make computers “speak” Lithuanian, it can be said that a major breakthrough occurred only in 2015, with the completion of the “Lithuanian Speech-Controlled Services” (LIEPA) project. During the LIEPA project, an electronic text reader (synthesizer) speaking in 4 voices was developed, along with 7 services that use it. For more details, see the LIEPA project website (https://www.raštija.lt/liepa). The algorithms, sound databases, and results achieved (speech intelligibility and naturalness) using Lithuanian speech synthesizers are described in greater detail in the article (Kasparaitis 2016). This paper discusses new speech-synthesis-based services developed following the LIEPA project, as well as services that could emerge in the near future. 1. EXISTING APPLICATIONS OF THE SYNTHESIZER The electronic text reader developed during the LIEPA project is freely distributed together with its source texts. This enables other people and organizations not involved in the LIEPA project to explore an ever-growing variety of ways to apply the synthesizer. 1.1. Performance Remote Vilnius For the first time, the LIEPA synthesizer voices Aistė and Edvardas were used while they were still unfinished. This happened at the international theatre festival Sirenos 2014, where there was PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 2 presented a performance specially created for Vilnius by the German troupe Rimini Protokoll, Remote Vilnius. The performance has neither a stage nor actors. The action takes place on specially selected streets of Vilnius or in other urban spaces; the scenery consists of houses, cars, and passers-by, while the actors—and at the same time the audience—are a group of about 50 people equipped with headphones, through which synthetic voices of Aistė or Edvardas (Juozas in the performance) sound, seeking to gain the listeners’ trust, engage them, and ultimately control this group of people. The performance seems to predict humanity’s future, when computers will lead people. A still from the performance is shown in Fig. 1; excerpts from the performance can be viewed in the LRT media library (FLM1 2014). The performance received numerous press reviews; a list of links can be found in the LIEPA project news (LN1 2014). The performance required voices that sounded mechanical and unnatural, so the synthesizer’s still-unpolished voices were perfectly suited to it. The voice recordings were also enhanced with various sound effects. FIG. 1. A still from the performance Remote Vilnius (photo by D. Matvejevas) BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 3 1.2. Audio narration of articles on websites Another obvious and therefore extensive application area is the audio narration of online articles, news items, and similar content. Its essence is perhaps best captured by the title of a Lietuvos žinios article (LŽ 2015): From now on, “Lietuvos žinios” can not only be read, but also listened to. In other words, an audio recording is published alongside the article text. The Lietuvos žinios portal (http://lzinios.lt/) and its mobile version (http://m.lzinios.lt/) were the first to start using a synthesizer for this purpose, on September 14, 2015; from the end of October 2015, Ūkininko patarėjas (http://ukininkopatarejas.lt/), also joined in; from February 2016, the Vilnius City Municipality website (http://www.vilnius.lt/) (VLN 2016), and from February 2017, the mobile version of the Delfi news portal (http://m.delfi.lt/) (DELFI 2017). The number of new articles appearing on these websites is quite large, so audio recordings generated from the text are not edited, as there would not be enough time, resources, and so on. It is very important that the synthesizer make as few errors as possible. The greatest difficulties for synthesizer developers arise during initial text processing, that is, converting numbers, dates, abbreviations, special characters, and other non-text information into text. Different websites may use different text preparation rules, so the synthesizer’s text-processing module needs to be adapted to each website. Audio recordings can also be provided alongside articles on websites with fixed content. The main difference from the application described above is that the number of articles is finite or changes slowly, so the text of each article can be adapted for the synthesizer, or two text versions can be prepared altogether: one for the reader and another for the synthesizer. The synthesizer can also be adapted somewhat to the text, for example by adding entries to the synthesizer’s abbreviation dictionary. An example of this application is the Visuotinė lietuvių enciklopedija (https://www.vle.lt/). The fragment on the Baltic peoples section (VLE 2017) clearly illustrates the kind of text presented to readers and how the synthesizer should revise it before reading: “…the Western Δ proto-language separated around the middle of the 1st millennium BC” “…the Proto-West Baltic language separated around the middle of the first millennium before Christ” At present, the synthesizer reads unprepared text, that is, it should itself determine which word to insert in place of the triangle, select the appropriate inflected form of the numeral, expand abbreviations, BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 4 to match word inflections, although it cannot yet do so. This example also contains another small problem that readers may not even notice—the hyphen-minus was used instead of a hyphen. 1.3. Text-to-speech service Text-to-speech services are most often used by blind and visually impaired people, but they can also be useful to other users, for example, when recording lecture notes for school pupils or students. One such service is RoboBraille, developed by Synscenter Refsnæs together with Sensus ApS of Denmark (http://robobraille.org/). The Lithuanian synthetic voices created by the LIEPA project have been available through RoboBraille since November 2015 (ROBO 2015, LN2 2015). RoboBraille is a service that automatically converts text documents into a range of alternative formats suitable for people with visual or reading disabilities. One of the available formats is an MP3 audio file. It is a self-service tool available 24 hours a day, 7 days a week. The service is free for individual non-commercial users. Users can either select one or more text files (available formats: .DOC,.DOCX,.PDF,.PPT,.PPTX,.TXT,.HTML,.HTM,.RTF,.EPUB,.MOBI, etc.), specify an internet address, or simply enter the desired text in the designated field. They can then choose one of four Lithuanian voices and one of seven speeds. Once the system has prepared an MP3 audio file, it sends a link to it to the email address provided by the user. 1.4. Bus station information Another application of a speech synthesizer is to read aloud announcements for passengers at bus and railway stations, airports, and so on. Thanks to the efforts of Algimantas Ankevičius, director of Vilkaviškis Bus Station UAB, this is already in operation. Voice recordings began replacing live announcements here in 1980, when a clock was connected to a tape recorder; in 2012, the recordings were replaced by a Lithuanian speech synthesizer, and since the second half of 2015, voices from the LIEPA project have been heard at Vilkaviškis Bus Station. For more details, see the video (FLM4 2015). From 7 a.m. to 7 p.m., passengers are given the exact time, bus arrival and departure information, and other announcements. Only announcements are read aloud on the platform, while announcements and radio programmes are broadcast in the passenger hall. An announcement has to be read within a minute, BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 5 because the radio switches on after one minute. Announcements are read by duets of synthesised voices (a female and a male voice take turns reading a few phrases each), much like television presenters. 1.5. ROBOT NAO FIG. 2. Robot NAO (http://old.mii.lt/index.php?siteaction=news_notices.view&id=4009&lang=lt) is a humanoid robot owned by the Robotikos akademijos and UAB Baltic Orbis that has learned to speak Lithuanian NAO (Fig. 2). It invited visitors to the Lithuanian stand during EU OPEN DAY 2016 in Brussels, where European Union countries presented themselves (LN3 2016, FLM2 2016). Linas Aidokas, an engineer at the Robotikos akademijos, installed the Lithuanian synthetic voices created by the LIEPA project in the robot and taught it to move; Prof. Laimutis Telksnys (VU MII), Dr. Rita Miliūnaitė (LKI), and Tautė Bernotaitė (LKI) prepared the robot’s performance script. The NAO robot also taught Lithuanian words to visitors at European Day of Languages events held at the Hermitage Museum in St. Petersburg (nearly 3,000 visitors came to the Lithuanian stand) (FLM3 2016); it recited an excerpt from Kristijonas Donelaitis’s Metų at the conference Can Robots Speak Lithuanian? held at the Seimas of the Republic of Lithuania (LN4 2016). Perhaps one of the LIEPA project’s applications that most captures the interest of young users of modern technologies BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 6 2. POTENTIAL APPLICATION EXAMPLES This section presents potential applications of speech synthesis that have not yet been implemented, although initial steps have already been taken toward them. 2.1. Porting the synthesizer to OS LINUX The LIEPA project synthesizer was developed for the Windows environment. Adapting it to the Linux environment would increase the number of ways it could be used. First and foremost, it could be used to provide audio for articles on online newspapers, magazines, and other similar websites, as already described in Section 1.2, but on websites built on the Linux platform. Similarly, the synthesizer could also be used to provide audio for websites with more stable content; for example, there are plans to provide an audio version of the encyclopedia Wikipedia (see Chirgwin 2016). 2.2. Virtual assistants As artificial intelligence advances, virtual assistants are being developed that can answer questions people phrase in their own words. Visitors to the website of the Migration Department under the Ministry of the Interior of the Republic of Lithuania (http://www.migracija.lt/) are assisted by the virtual assistants Martynas and Ieva. Users can ask them questions, and they answer or provide a list of links to relevant pages on the website. For example, the question How much does a passport cost will receive the following answer: Fixed state fees for issuing personal identity documents: • Passport; • Identity card; • temporary citizen's certificate. Fee exemption How to pay Please note that personal identity documents are issued by the migration units of territorial police institutions. BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 7 Another page opens List of State Fees. The same answer is also given to the question How much does an identity card cost; that is, the answers are prepared in advance and are somewhat more general than the questions. For inappropriate questions, the assistant provides responses of this kind: Please clarify your question; I will try to help you if you provide short questions; I will try to answer your question within the competence of the Migration Department. One drawback of virtual assistants is that their responses are not voiced in Lithuanian. FIG. 3. Virtual assistants Ieva and Martynas (http://www.migracija.lt/) BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERYS. Lietuviško balso sintezės dabartis ir perspektyvos | 8 2.3. ROBOT MIO MOBILE KIOSK Many holidaymakers on the Palanga coast have seen young people carrying coolers and constantly calling out: “Hot chebureki! Fried bread! …” A robot could do this job too. One Lithuanian start-up is currently developing an autonomous mobile vending machine designed specifically for this purpose, called MIO mobile kiosk (Fig. 4) (http://miokiosk.com/) (Fuks 2016). It is expected to be able to carry and sell snacks and drinks in parks, squares, and plazas, as well as during various events. The plan is to make this robot available to summon via a mobile app, and to use a touchscreen to communicate with customers. For users’ convenience, the robot could also speak Lithuanian: call out to customers, advertise its products, provide instructions for use, and so on. FIG. 4. MIO mobile kiosk (http://miokiosk.com/) 2.4. Machine Translation Programs The first computer programs capable of translating text from one language to another appeared several decades ago. Some of them can read aloud the text being translated or already BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 9 translated text. For example, one of the most popular programs, the Google translator (http://translate.google.lt/), can read text written in the languages of neighboring countries: Polish, Russian, and Latvian. At least four translation programs that translate into or from Lithuanian are known: the aforementioned Google translator, the Microsoft translator (http://www.bing.com/translator/),, the Vytautas Magnus University translator (http://vertimas.vdu.lt/twsas/),, Tildė IT, and the Institute of the Lithuanian Language translator (https://translate.tilde.com/lt/).. Unfortunately, these translation programs cannot read Lithuanian text aloud. 2.5. Adaptation of smartphones for blind people In recent years, many tasks that people used to perform on computers, such as browsing the internet or writing emails, are now performed on smartphones. Blind and visually impaired people are no exception; they too are increasingly using smartphones. As with computers, feedback is provided to blind users by voice. For this purpose, smartphones have built-in speech synthesizers and screen readers for blind users connected to them through a standardized interface. The most popular smartphone operating systems and their built-in readers are: the open-source system Android and the TalkBack reader; Apple iOS and VoiceOver; and Microsoft Windows Mobile and Narrator. Unfortunately, neither Apple nor Microsoft has developed a Lithuanian synthesizer, and the ability to install a synthesizer of one’s own on their devices is specifically restricted. Therefore, at present, only Android smartphones can be made to speak Lithuanian. To do this, the synthesizer software would need to be ported to the Android system and a standard interface created; blind users could then use the smartphone with the help of the built-in TalkBack screen reader. Several obstacles are associated with porting the LIEPA synthesizer to the Android system: Smartphones generally have less memory than computers, so the sound databases would need to be reduced, which could degrade voice quality. However, smartphones may be able to use a lower signal sampling rate, making it possible to reduce the sound databases simply by lowering the rate; In the Android system, the interface is usually programmed in the Java programming language, which is an interpreted language, whereas the synthesizer is written in C, which is BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos |10 compiled; fortunately, the Android system allows C functions to be called from Java, and vice versa; • The synthesizer must be able to quickly interrupt the reading of a phrase already started and move on to the next phrase; • The Android system is used by many manufacturers across different phone models. Menu items, notifications, abbreviations used in them, and so on may differ. For example, the Samsung Galaxy J3 phone has a menu item Aut. raš. didž. raidėmis, which should be read as Automatically capitalize. Special adaptation would be required for as many different models as possible. 2.6. Car navigation After porting the LIEPA project’s speech synthesizer to the Android environment and creating a standard Android interface for it, other applications that understand this interface could also use the synthesizer. One of them is the iGO Primo navigation app running on a smart Android phone. Most navigation systems use voice recordings, and only a few synthesize speech. Navigacijos sistemos paprastai atlieka šias funkcijas (NAV 2017): įvedus adresą ar žemėlapyje nurodžius vietą, suskaičiuoja ir žemėlapyje parodo visą maršrutą, būsimos kelionės trukmę ir atstumą. Važiuojant priartina žemėlapį, šneka ir rodo, kaip važiuoti. Balsu pasako iš anksto, vėliau kelis kartus pakartoja, kai reikia pasukti, įvažiuoti, išvažiuoti, persirikiuoti ir kt. Klaidingai pasukus perskaičiuoja maršrutą. Perspėja apie svarbias kelionės vietas: valstybės sienos kirtimą, greičio matuoklius, maksimalų leidžiamą greitį ir ar jis neviršytas ir pan. Difficulties in adapting the synthesizer for navigation: • The navigation app obtains information from various sources, such as maps in several different formats, speed camera lists, and so on. It must be able to distinguish information that should not be read aloud; • Navigation messages are generated dynamically from templates written in configuration files; these templates can be combined, nested within one another, and filled with other information. BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 11 The system must be able to correctly modify numbers and abbreviations in a sentence assembled from templates, inflect them appropriately, and so on; for example, Turn left in 3 km 500 m. SUMMARY By the end of the LIEPA project, a major leap had been achieved in the quality of Lithuanian speech synthesis. More importantly, however, people and organizations began taking the initiative to seek new applications for speech synthesis. The synthesizer has already been used in a theatrical performance, to provide voice-over for articles on online magazines, encyclopedias, and other websites, to read information announcements at a bus station, and as a text-to-speech service; the robot NAO has also been made to speak Lithuanian. New applications can also be expected in the near future. To broaden the range of applications further, the synthesizer should be ported to other environments, such as Linux computers and Android smartphones. Another challenge that often arises when developing a new application is improving the initial text-processing module or preparing a large volume of texts for the synthesizer. ŠALTINIAI Chirgwin R. 2016, Wikipedia to build and give away speech synthesis code. A talking Wikipedia? Yes please, say the visually-impaired. Prieiga internete: http://www.theregister.co.uk/2016/03/14/wikipedia_seeks_help_for_speech_synthesis/, (žiūrėta 2017 09 01). DELFI 2017: Mobiliajame DELFI – galimybė klausytis naujienas. Prieiga internete: http://www.delfi.lt/m360/naujausi-straipsniai/mobiliajame-delfi-galimybe-klausytis-naujienas.d?id=73617338, (žiūrėta 2017 09 01). FLM1 2014: Teatras. Pasivaikščiojimas po „Sirenas“ su trupėmis „Rimini Protokoll“ ir „Teatr Post“. Prieiga internete: http://www.lrt.lt/mediateka/irasas/65190/teatras nuo 19:50 iki 32:40, (žiūrėta 2017 09 01). FLM2 2016: Lietuviškai šnekantis NAO robotas Briuselyje. Prieiga internete: https://www.youtube.com/watch?v=mzkBIhLZxC0, (žiūrėta 2017 09 01). BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos | 12 FLM3 2016: Lietuviškai šnekantis robotas konsulate. Prieiga internete: https://www.youtube.com/watch?v=kYkPBRlxiCs, (žiūrėta 2017 09 01). FLM4 2015: Vilkaviškio autobusų stotis. 2015. Prieiga internete: https://www.youtube.com/watch?v=sS9xKZpVeGU, (žiūrėta 2017 09 01). Fuks E. 2016: Mobilus kioskas: gali atvežti, ko norite. Prieiga internete: http://www.delfi.lt/projektai/archive/mobilus-kioskas-gali-atvezti-ko-norite.d?id=73308794, (žiūrėta 2017 09 01). Google vertyklė. Prieiga internete: http://translate.google.lt/, (žiūrėta 2017 09 01). Kasparaitis P. 2016: Lietuviško balso sintezatorių kokybės vertinimas. Kalbų studijos / Studies about languages. 28, 80–91. Prieiga internete: http://kalbos.ktu.lt/index.php/KStud/article/view/15130, (žiūrėta 2017 09 01). LIEPA svetainė. Prieiga internete: https://www.raštija.lt/liepa, (žiūrėta 2017 09 01). Lietuvos žinių portalas. Prieiga internete: http://lzinijos.lt/, (žiūrėta 2017 09 01). Lietuvos žinių portalo mobilioji versija. Prieiga internete: http://m.lzinios.lt/, (žiūrėta 2017 09 01). LN1 2014: Projekto LIEPA sintezatorius „vaidino“ spektaklyje „Remote Vilnius“ (nuorodos) Prieiga internete: https://www.raštija.lt/naujienos/projekto-liepa-naujienos/projekto-liepa-sintezatorius-vaidino-spektaklyje-remote-vilnius-nuorodos/223?partner=11, (žiūrėta 2017 09 01). LN2 2015: Projekto LIEPA sintetiniai balsai - svetainėje Robobraille.org. Prieiga internete: https://www.raštija.lt/naujienos/projekto-liepa-naujienos/projekto-liepa-sintetiniai-balsai--svetaineje-robobrailleorg/7613, (žiūrėta 2017 09 01). LN3 2016: Lietuviškai šnekantis robotas EU OPEN DAY renginyje. Prieiga internete: https://www.raštija.lt/naujienos/projekto-liepa-naujienos/lietuviškai-šnekantis-robotas-eu-open-day-renginyje/7635, (žiūrėta 2017 09 01). LN4 2016: Robotas prašneko lietuviškai Lietuvos Respublikos Seime. Prieiga internete: https://www.raštija.lt/naujienos/projekto-liepa-naujienos/robotas-prašneko-lietuviškai-lietuvos-respublikos-seime/7632, (žiūrėta 2017 09 01). LŽ 2015: Nuo šiol „Lietuvos žinios“ ne tik skaitomos, bet ir klausomos. Prieiga internete: http://lzinijos.lt/lzinijos/Lietuva/nuo-siol-lietuvos-zinios-ne-tik-skaitomos-bet-ir-klausomos/208668, (žiūrėta 2017 09 01). BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos |13 Microsoft vertyklė. Prieiga internete: http://www.bing.com/translator/, (žiūrėta 2017 09 01). Migracijos departamentas prie Lietuvos Respublikos vidaus reikalų ministerijos. Prieiga internete: http://www.migracija.lt/, (žiūrėta 2017 09 01). MIO mobile kiosk. Prieiga internete: http://miokiosk.com/, (žiūrėta 2017 09 01). NAV 2017: Kaip kas veikia / Navigacinės savybės. Prieiga internete: http://www.euronet.lt/produktai/veikimo-principas/navigacines-savybes, (žiūrėta 2017 09 01). ROBO 2015: Four New Lithuanian Voices in RoboBraille. Prieiga internete: http://robobraille.org/news/four-new-lithuanian-voices-robobraille, (žiūrėta 2017 09 01). Tildė IT ir Lietuvių kalbos instituto vertyklė. Prieiga internete: https://translate.tilde.com/lt/, (žiūrėta 2017 09 01). Ūkininko patarėjas. Prieiga internete: http://ukininkopatarejas.lt/, (žiūrėta 2017 09 01). Vytauto Didžiojo universiteto vertyklė. Prieiga internete: http://vertimas.vdu.lt/twsas/, (žiūrėta 2017 09 01). VLE 2017: Visuotinė lietuvių enciklopedija: baltai. Prieiga internete: https://www.vle.lt/PaieskosRezultatai/ctl/Straipsnis/mid/469/tid/68466, (žiūrėta 2017 09 01). VLN 2016: www.vilnius.lt – pirmoji iš Lietuvos savivaldybių svetainių, naudojanti teksto skaitytuvą. Prieiga internete: (http://www.vilnius.lt/index.php?2914688187, (žiūrėta 2017 09 01). Gauta 2017 09 06 PIJUS KASPARAITIS, GINTARAS SKERSYS Vilniaus universitetas, Matematikos ir informatikos fakultetas Didlaukio 47, VILNIUS, Lietuva [email protected], [email protected] BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204 PIJUS KASPARAITIS, GINTARAS SKERSYS. Lietuviško balso sintezės dabartis ir perspektyvos |14 Pijus Kasparaitis, Gintaras Skersys PRESENT AND PERSPECTIVES OF THE LITHUANIAN TEXT-TO-SPEECH SYNTHESIS Summary Lithuanian text-to-speech synthesis is introduced: where it is currently used, where it is intended to be used in the future, what are outstanding unresolved problems and how the philologists could help informatics to solve them. BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204