Development of the Feature Extractor for Speech Recognition

ENEKO AÑORGA DEVELOPMENT OF THE FEATURE EXTRACTOR FOR SPEECH RECOGNITION DIPLOMA WORK MARIBOR, OCTOBER 2009 i FAKULTETA ZA ELEKTROTEHNIKO, RAČUNALNIŠTVO IN INFORMATIKO 2000 Maribor, Smetanova ul. 17 Diploma Work for Electronic Engineering Student Program DEVELOPMENT OF THE FEATURE EXTRACTOR FOR SPEECH RECOGNITION Student: Eneko A ñ orga Student program: Electronic Engineering Mentor: Prof. Dr. Riko ŠAFARIČ Asist. Prof. Dr. Suzana URAN Maribor, October 2009 ii ACKNOWLEDGMENTS Thanks to Prof. Dr. Riko ŠAFARIČ for his assistance and helpful advices in carrying out the diploma work. Special thanks to my family and friends who are in all moments beside me. iii DEVELOPMENT OF THE FEATURE EXTRACTOR FOR SPEECH RECOGNITION Key words: voice operated wheelchair, speech recognition, voice activity detection, neural networks, ultrasound sensor net UDK: 004.934:681.5(043.2) Abstract With this diploma work we have attempted to give continuity to the previous work done by other researchers called, Voice Operating Intelligent Wheelchair – VOIC [1]. A development of a wheelchair controlled by voice is presented in this work and is designed for physically disabled people, who cannot control their movements. This work describes basic components of speech recognition and wheelchair control system. Going to the grain, a speech recognizer system is comprised of two distinct blocks, a Feature Extractor and a Recognizer. The present work is targeted at the realization of an adequate Feature Extractor block which uses a standard LPC Cepstrum coder, which translates the incoming speech into a trajectory in the LPC Cepstrum feature space, followed by a Self Organizing Map, which classifies the outcome of the coder in order to produce optimal trajectory representations of words in reduced dimension feature spaces. Experimental results indicate that trajectories on such reduced dimension spaces can provide reliable representations of spoken words. The Recognizer block is left for future researchers. The main contributions of this work have been the research and approach of a new technology for development issues and the realization of applications like a voice recorder and player and a complete Feature Extractor system. iv RAZVOJ PREVODNIKA SIGNALA ZA PREPOZNAVO GOVORA Ključne besede: glasovno voden invalidski voziček, prepoznava govora, zaznava glasovne aktivnosti nevronske mreže, ultrazvočna senzorska mreža UDK: 004.934:681.5(043.2) Povzetek S tem diplomskim delom sem poskusil nadaljevati delo raziskave z naslovom Voice Operating Intelligent Wheelchair – VOIC [1]. V tem diplomskem delu je tudi predstavljen razvoj glasovno vodenega invalidskega vozička, narejenega za telesno prizadete ljudi, ki ne morejo nadzorovati svojih gibov. To delo opisuje osnovne komponente govornega nadzora in sistema vodenja invalidskega vozička. Sistem govornega nadzora uravnavata dva različna dela; prevodnik signala in prepoznavalec. V tem diplomskem delu se osredotočam na prevodnost ustreznega prevodnika signala na osnovi standardnega LPC Cepstrum koderja, ki posreduje prihajajoči govor v pot LPC Cepstrum prostora, temu postopku pa sledi t.i. “samoorganizacijska karta” (Self Organizing Map), ki razvrsti rezultat koderja za optimalni prikaz besed na zmanjšanih dimenzijah prostora. Poskusni rezultati kažejo, da lahko te poti na zmanjšanih dimenzijah prostora zagotovijo zanesljiv prikaz izgovorjenih besed. Prepoznavalec je lahko predmet raziskovanja študentov tudi v prihodnosti. Glavni namen tega diplomskega dela sta bili raziskava in poskus uporabe nove tehnologije v razvojne namene, kakor tudi uporaba aplikacij kot so snemalec in predvajalnik zvoka ter celoten sistem prevodnika signala. v Table of Contents 1 INTRODUCTION ........................................................................................................ 1 1.1 MOTIVATION ..................................................................................................... 1 1.2 OBJECTIVES AND CONTRIBUTION OF THIS WORK.................................. 3 1.3 ORGANIZATION OF THIS WORK ................................................................... 5 1.4 RESOURCES ........................................................................................................ 5 2 DEVELOPMENT ......................................................................................................... 6 2.1 BRIEF DESCRIPTION OF COLIBRI MODULE ............................................... 6 2.1.1 Hardware ......................................................................................................... 6 2.1.2 Software ........................................................................................................ 10 2.2 DESIGN OF THE FEATURE EXTRACTOR ................................................... 11 2.2.1 Speech coding ............................................................................................... 12 2.2.1.1 Speech sampling ..................................................................................... 12 2.2.1.2 Pre-emphasis filter.................................................................................. 13 2.2.1.3 Word Isolation ........................................................................................ 13 2.2.1.4 Speech coding ........................................................................................ 14 2.2.1.5 Summing up ........................................................................................... 18 2.2.2 Dimensionality reduction using SOM ........................................................... 19 2.2.2.1 Optional Signal Scaling .......................................................................... 22 2.2.2.2 Summing up ........................................................................................... 22 2.3 SOFTWARE DEVELOPMENT ......................................................................... 24 2.3.1 Audio Recorder and Player ........................................................................... 24 2.3.2 Feature Extractor ........................................................................................... 27 3 CONCLUSIONS ........................................................................................................ 33 3.1 SUMMARY OF RESULTS ................................................................................ 33 3.2 DIRECTIONS FOR FUTURE RESEARCH ...................................................... 34 4 REFERENCES ........................................................................................................... 36 vi GLOSARY OF SIMBOLS Name Description 1/Â(z) a i w lp (n) H h1 (z) s(n) s'(n) r(k) w lag (n) r'(k) k i f s f 0 LP synthesis filter LP coefficients (a 0 = 1.0) LP analysis window Input high-pass filter Preprocessed/filtered speech signal Windowed speech signal Auto-correlation coefficients Correlation lag window Modified auto-correlation coefficients Reflection coefficients Sampling frequency Bandwidth expansion Table 1 – Glossary of symbols vii GLOSSARY OF ACRONYMS Acronym Description VOIC DSP LPC LP SOM CE SODIMM CAN GPIO BSP ITU ITU-T FE VAD DTX CNG RNN VQ HMM Voice Operated Intelligent Wheelchair Digital Signal Processor Linear Prediction Coding Linear Prediction Self Organizing Maps Compact Edition Small Outline Dual In-line Memory Module Control Area Network General Purpose Input/Output Board Support Package International Telecommunication Union Telecommunication Standardization Sector Feature Extractor Voice Activity Detection Discontinuous Transmission Comfort Noise Generator Recurrent Neural Network Vector Quantization Hidden Markov Model Table 2 – Glossary of acronyms Development of the Feature Extractor for Speech Recognition 7 Figure 2.1 – Colibri PXA320 Module Specifications: CPU PXA320 806MHz Memory 128MB DDR RAM (32Bit) 1GB NAND Flash (8Bit) Interfaces 16Bit External BUS Compact Flash/PCMCIA LCD (SVGA) Touch Screen Audio I/O (16Bit Stereo) CMOS image sensor I2C SPI 2x SD Card USB Host/Device 100MBit Ethernet 2x UART IrDA PWM 127 GPIOs Software Pre-installed Windows CE 5.0/6.0 Size 67.6 x 36.7 x 5.2 mm Temperature Range 0 to +70°C -45 to +85°C (IT version) 8 Eneko Añorga, Diploma Work In order to have a flexible development environment to explore the functionality and performance of the Colibri modules the Colibri Evaluation Board is used (See Figure 2.2). Besides the user interfaces it provides numerous communication channels as well as a configurable jumper area to hook up the Colibri GPIOs to the desired function. To facilitate interfacing to the custom hardware the Colibri Evaluation Board provides the buffered CPU bus on a separate connector. Development of the Feature Extractor for Speech Recognition 9 Figure 2.2 – Colibri Evaluation Board Module Specifications: CPU Modules Colibri PXA270 Colibri PXA300 Colibri PXA310 Colibri PXA320 Interfaces 10/100MBit Ethernet USB Host/Device USB Host 2x PS/2 Analogue VGA Generic LCD Connector TFT: Philips LB064V02-A1 Line-In, Line-Out, Mic-In IrDA 2x RS232 CAN (Philips SJA1000) SD Card Compact Flash Power Supply: Required Input: 7-24VDC, 3-50W On-board Converter: 3.3V, 5V max 5A Size: 200 x 200 mm 10 Eneko Añorga, Diploma Work The received invoice from Toradex, for the Colibri XScale® PXA320, plus the Colibri Evaluation Carrier Board and plus the support hours are shown in the next Figure 2.3: Figure 2.3 – Invoice from Toradex 2.1.2 Software The module is shipped with a preinstalled WinCE 5.0 image with WinCE Core license. Other OS like Embedded Linux are available from the third-party. Toradex provides a WinCE 5.0 image and a WinCE 6.0. All WinCE images contain the Toradex Board Support Package (BSP) which is one of the most advanced BSPs available on the market. Besides the standard Windows CE functionality, it includes a large number of additional drivers as well as optimized versions of standard drivers for the most common interfaces and is easily customizable by registry settings to adapt to specific hardware. The Microsoft® eMbedded Visual C++ 4.0 tool is used as desktop development environment for creating the applications and system components for Windows® CE .NET powered devices. In conclusion, all the software presented in this work was done using the Microsoft® eMbedded Visual C++ 4.0 development tool and the Toradex BSP tool (Figure 2.4). Development of the Feature Extractor for Speech Recognition 11 Figure 2.4 – Microsoft® eMbedded Visual C++ 4.0 and Windows® CE 2.2 DESIGN OF THE FEATURE EXTRACTOR As stated before, in a speech recognition problem the FE block has to process the incoming information, the speech signal, so that its output eases the work of the classification stage. The approach used in this work designs the FE block and divides it into two consecutive sub-blocks: the first is based on speech coding techniques, and the second uses a SOM for further optimization (data dimensionality reduction). The different blocks and sub-blocks are shown in the next Figure 2.5: 12 Eneko Añorga, Diploma Work Figure 2.5 – FE schematic 2.2.1 Speech coding 2.2.1.1 Speech sampling The speech was recorded and sampled using a relatively inexpensive dynamic microphone and a Colibri’s audio input interface. The incoming signal was sampled at 8.000 Hz with 16 bits of resolution. It might be argued that a higher sampling frequency, or more sampling precision, is needed in order to higher recognition accuracy. However, if a normal digital phone, which samples speech at 8.000 Hz with a 16 bit resolution, is able to preserve most of the information carried by the signal [6], it does not seem necessary to increase the sampling rate beyond 8.000 Hz or the sampling precision to something higher than 16 bits. Another reason behind these settings is that Development of the Feature Extractor for Speech Recognition 13 commercial speech recognizers typically use comparable parameter values and achieve impressive results. 2.2.1.2 Pre-emphasis filter After sampling the input signal is convenient to filter it with a second order high-pass filter with cut off frequency at 140 Hz. The filter serves as a precaution against undesired lowfrequency components. The resulting filter is given by: ( ) 21 21 1 9114024 . 0 9059465 . 1 1 46363718.092724705.046363718.0 −− −− + − +− = z z zz zH h (1) 2.2.1.3 Word Isolation Despite the fact that the sampled signal had pauses between the utterances, it was still needed to determine the starting and ending points of the word utterances in order to know exactly the signal that characterized each word. To accomplish this, we decided to use VAD (Voice Activity Detection) technique used in speech processing, instead of using the rolling average and the threshold, determined by the start and end of each word, used in the previous works, with the aim of achieving more accuracy and efficiency. For that, we based our work in the Annex B from the ITU’s Recommendation G.729 [5], where a source code in C language about the VAD is efficiently developed. VAD is a method which differentiates speech from silence or noise signal to aid in speech processing and the Annex B provides a high level description of the Voice Activity Detection (VAD), Discontinuous Transmission (DTX) and Comfort Noise Generator (CNG) algorithms. These algorithms are used to reduce the transmission rate during silence periods of speech. They 14 Eneko Añorga, Diploma Work are designed and optimized to work in conjunction with [ITU-T V.70]. [ITU-T V.70] mandates the use of speech coding methods. The algorithms are adapted to operate with both the full version of G.729 and Annex B. Let’s see a general description of the VAD algorithm: The VAD algorithm makes a voice activity decision every 10 ms in accordance with the frame size of the pre-processed (filtered) signal. A set of difference parameters is extracted and used for an initial decision. The parameters are the full-band energy, the low-band energy, the zero-crossing rate and a spectral measure. The long-term averages of the parameters during nonactive voice segments follow the changing nature of the background noise. A set of differential parameters is obtained at each frame. These are a difference measure between each parameter and its respective long-term average. The initial voice activity decision is obtained using a piecewise linear decision boundary between each pair of differential parameters. A final voice activity decision is obtained by smoothing the initial decision. The output of the VAD module is either 1 or 0, indicating the presence or absence of voice activity respectively. If the VAD output is 1, the G.729 speech codec is invoked to code/decode the active voice frames. However, if the VAD output is 0, the DTX/CNG algorithms described herein are used to code/decode the non-active voice frames. 2.2.1.4 Speech coding After the signal was sampled, the spectrum was flattened, and the utterances were isolated we tried to codify it using the Linear Prediction Coding (LPC) method [3]. In a variety of applications, it is desirable to compress a speech signal for efficient transmission or storage. For example, to accommodate many speech signals in a given bandwidth of a cellular phone system, each digitized speech signal is compressed before transmission. For medium or low bit-rate speech coders, LPC method is most widely used. Redundancy in a speech signal is removed by passing the signal through a speech analysis filter. Development of the Feature Extractor for Speech Recognition 15 The output of the filter, termed the residual error signal, has less redundancy than the original speech signal and can be quantized by a smaller number of bits than the original speech. The short-term analysis and synthesis filters are based on 10th order linear prediction (LP) filters. The LP synthesis filter is defined as: ∑ = − + = 10 1 ˆ 1 1 )( ˆ1 i i i za zA (2) where â i , i = 1,...,10, are the quantized Linear Prediction (LP) coefficients. Short-term prediction or linear prediction analysis is performed once per speech frame using the autocorrelation method with a 30 ms (240 samples) asymmetric window. Every 10 ms (80 samples), the autocorrelation coefficients of windowed speech are computed and converted to the LP coefficients using the Levinson-Durbin algorithm. Then the LP coefficients are transformed to the LSP domain for quantization and interpolation purposes. The interpolated quantized and unquantized filters are converted back to the LP filter coefficients (to construct the synthesis and weighting filters for each subframe). The LP analysis window consists of two parts: the first part is half a Hamming window and the second part is a quarter of a cosine function cycle. The window is given by: ( ) ( ) 239...,002 159 2002 cos 0,...,199 399 2 cos 46.054.0        =      − =       − = n n n n nw lp π π (3) There is a 5 ms look-ahead in the LP analysis which means that 40 samples are needed from the future speech frame. This translates into an extra algorithmic delay of 5 ms at the encoder stage. The LP analysis window applies to 120 samples from past speech frames, 80 samples from 16 Eneko Añorga, Diploma Work the present speech frame, and 40 samples from the future frame. The windowing procedure is illustrated in Figure 2.4. Figure 2.6 – Windowing procedure in LP analysis The different shading patterns identify corresponding excitation and LP analysis windows. The windowed speech: ( ) ( ) ( ) 0,...,239 = = ′ nnsnwns lp (4) is used to compute the autocorrelation coefficients: ( ) ( ) ( ) 0,...,10 239 =−= ∑ = kkn'sn'skr kn (5) To avoid arithmetic problems for low-level input signals the value of r(0) has a lower boundary of r(0) = 1.0. A 60 Hz bandwidth expansion is applied by multiplying the autocorrelation coefficients with: ( ) 1,...,10 2 2 1 2 0 =                π −= k f kf expkw s lag (6) Development of the Feature Extractor for Speech Recognition 23 1. The incoming pressure wave is reduced to a digital signal through a sampling process. 2. The starting and the ending points of the utterance embedded into the signal are obtained using the VAD process. 3. The spectrum of the extracted utterance is enhanced by means of a pre-emphasis filter which boosts the high frequency components. 4. Several data blocks are extracted from the enhanced signal. 5. The extracted data blocks are windowed to reduce leakage effects. 6. LPC components are extracted from the filtered blocks. 7. LPC Cepstrum components are then extracted from the LPC vectors. 8. The dimensionality of the LPC Cepstrum vectors is reduced using a SOM. 9. The resulting vectors are scaled if the Recognizer requires it. Nothing can still be said about the overall effectiveness of the FE block, since it depends on the recognition accuracy of the overall system. As an example, if the complete system achieves low recognition percentages, that can be caused by the FE block or the Recognizer, but, if it achieves higher percentages that means that the FE block was at least able to produce data that allowed these recognition accuracies. In other words, in order to know the usefulness of the FE block, the Recognizer outputs must be obtained first. 24 Eneko Añorga, Diploma Work 2.3 SOFTWARE DEVELOPMENT 2.3.1 Audio Recorder and Player As we have mentioned before, we built a complete audio Recorder and Player. This task was not strictly necessary but the aim of this has been to learn, practice and improve the C ++ programming skills. The graphic interface of the program is shown in the following Figure 2.8: Figure 2.8 – audioce Recorder & Player Development of the Feature Extractor for Speech Recognition 25 As we can see in the Figure 2.8, with this software we are able to record the voice during some time, with different sample rates, different resolutions and finally we can save it in a .wav file for next processing steps like the speech coding. In addition, we will be able to play and listen to the recorded signals. Along with the documentation (Annex 2) is included an electronic attachment containing the source code in C++ used to build the software and which is coming with all the necessary explanations. As we can see in the next Figure 2.9 we can see the difference between the different ways of sampling. In general, the memory occupied by the sound file is proportional to the number of samples per second and the resolution of each sample. For the first case the speech is sampled at 8.0 KHz and with 8 bits of resolution (1 byte/sample), this means that the memory occupied for the sound file will be 8.000 (samples/sec) x 5 (sec) = 40.000 (samples) = 40.000 (samples) x 1 (byte/sample) = 40 Kbytes. For the second case the speech is sampled at 44.1 KHz and with 16 bits of resolution (2 bytes/sample), the memory occupied for the sound file will be 44.100 (samples/sec) x 5 (sec) = 220.500 (samples) = 220.500 (samples) x 2 (bytes/sample) = 440.1 Kbytes. 26 Eneko Añorga, Diploma Work Figure 2.9 – Different samplings of the same speech signal It might be argued that the higher the sampling frequency and the higher the sampling precision, the better the recognition accuracy. However, if a normal digital phone, which samples speech at 8.000 Hz with a 16 bit precision, is able to preserve most of the information carried by the signal [6], it does not seem necessary to increase the sampling rate beyond 8.000 Hz or the sampling precision to something higher than 16 bits. Another reason behind these settings is that commercial speech recognizers typically use comparable parameter values and achieve impressive results. So for the speech coder we decided to use 8.000 Hz of sample rate and 16 bits of resolution as configuration to record the voice. Development of the Feature Extractor for Speech Recognition 27 2.3.2 Feature Extractor After built the audioce Recorder and Player we added some new functions to the program as the part which represents the FE block. For that, we based our work on the ITU-T’s G.729 Recommendation. This Recommendation contains the description of an algorithm for the coding of speech signals using Linear Prediction Coding and in which more processes like the filtering of the sampled signal, the division into blocks and the windowing of filtered signal and the word isolation are implicit. The graphic interface of the program is shown in the following Figure 2.10. Figure 2.10 – audioce Recorder & Player & VAD Detector & Speech Coder 28 Eneko Añorga, Diploma Work With this software we are able to record and play voice signals and save them in .wav files. In addition, we will be able to detect and isolate the voice command from the sound file, create a new file with it and finally codify to reduce its dimensionality. Along with the documentation (Annex 2) is included an electronic attachment containing the source code in C++ used to build the software and which is coming with all the necessary explanations. When we press the button “VAD Detector...” we have to select a sound file which contains the recorded voice. Then the sampled signal will go through all these steps: (Sampling: the voice signal is sampled at 8.000 Hertz with 16 bits of precision and saved in a new file called “left.wav”.) 1. Pre-emphasis filter: the sampled signal is filtered by a second order high-pass filter and saved in a new file called “filtered.wav”. 2. Word Isolation: the filtered signal is passed through the VAD block to isolate the word and saved in a file called “vad.wav”. We can represent the results of the different stages of the signal using Matlab and the earlier created sound files (Figure 2.11). Development of the Feature Extractor for Speech Recognition 29 Figure 2.11 – Different stages of the signal in VAD Detector process As we can see, the first graphic shows the original signal (Command “right”), sampled at 8.0 KHz, with 16 bits of resolution and in mono or 1 channel. We can realize that the signal has a DC offset and also that the end of the recording, is a bit noisy. The second graphic shows the filtered sampled signal and finally the third graphic shows the isolated word, after the VAD process. At this point we have to explain that the result shown for the VAD process was obtained using an algorithm myVAD.m [10] obtaining pretty good results. The problem with the algorithm developed for Colibri module is that it does not avoid the silence and the non-speech parts from the voice very well as the myVAD.m algorithm does, as is shown in the next Figure 2.12. 30 Eneko Añorga, Diploma Work Figure 2.12 – Different ways of VAD process In the above Figure 2.12, the second graphic shows the recorded signal after VAD process using the algorithm developed for Colibri module. Here, we can see that even though the sound file is reduced in some silent parts of the signal are avoided; these process is not doing the correct work as the third graphic does. We tried to find the solution to this but finally we did not, so this part need to be improved and is left for future researchers as we explain in the section 3.2 Directions for future research. To continue our research we decided to use the file created trough Matlab and myVAD.m algorithm called “vad2.wav”. Once we get the isolated word (“vad2.wav”) we can start codifying the speech clicking in “Speech Coder...” button. The next steps are the ones which the isolated word signal will follow: Development of the Feature Extractor for Speech Recognition 31 1. Blocking: the isolated word is divided into a sequence of data blocks of fixed length, called frames and multiplied by Hamming window of same width. 2. LPC analysis: for each frame, 10 LPC coefficients are calculated. 3. Cepstrum analysis: the 10 LPC coefficients are converted in 10 Cepstral coefficient ones (Figure 2.13). Figure 2.13 – The 10 Cepstral coefficients for each frame 32 Eneko Añorga, Diploma Work Once we get the 10 Cepstral coefficients for each frame if we click in “SOM...” button and select the file with the Cepstral coefficients (“c_coeff.txt”), we will fed the input layer of the Kohonen SOM, with this input data one by one. The output layer will organize itself to represent the inputs in two-dimensional space. The training procedure involves the following steps: 1. The neurons are arranged in an n-dimensional lattice. Each neuron stores a point in an mdimensional space. 2. An input vector is presented to the SOM. The neurons start to compete until the one that stores the closest point to the input vector prevails. Once the dynamics of the network converge, all the neurons but the prevailing one will be inactive. The output of the SOM is defined as the co-ordinates of the prevailing neuron in the lattice. 3. A neighbourhood function is centred on the prevailing neuron of the lattice. The value of this function is one at the position of the active neuron, and decreases with the distance measured from the position of the winning neuron. 4. The points stored by all the neurons are moved towards the input vector in an amount proportional to the neighbourhood function evaluated in the position of the lattice where the neuron being modified stands. 5. Return to 2, and repeat steps 2, 3, and 4 until the average error between the input vectors and the winning neurons reduces to a small value. After the SOM is trained, the co-ordinates of the active neuron in the lattice are used as its outputs. Along with the documentation (Annex 2) is included an electronic attachment containing the source code in C++ used to build the software and which is coming with all the necessary explanations. In this case part of the code for SOM has been developed but still needs to be improved and finished, so this part is left for future researchers as we explain in the section 3.2 Directions for future research. Development of the Feature Extractor for Speech Recognition 39 ANNEX 2: 40 Eneko Añorga, Diploma Work DECLARATION: I, Eneko Añorga, the undersigned, declare that I have made the diploma work by myself. I am aware of the potential consequences in the event of a breach of this declaration.