Broad phonetic class definition driven by phone confusions

RESEARCH Open Access Broad phonetic class definition driven by phone confusions Carla Lopes 1,2* and Fernando Perdigão 1,3 Abstract Intermediate representations between the speech signal and phones may be used to improve discrimination among phones that are often confused. These representations are usually found according to broad phonetic classes, which are defined by a phonetician. This article proposes an alternative data-driven method to generate these classes. Phone confusion information from the analysis of the output of a phone recognition system is used to find clusters at high risk of mutual confusion. A metric is defined to compute the distance between phones. The results, using TIMIT data, show that the proposed confusion-driven phone clustering method is an attractive alternative to the approaches based on human knowledge. A hierarchical classification structure to improve phone recognition is also proposed using a discriminative weight training method. Experiments show improvements in phone recognition on the TIMIT database compared to a baseline system. Introduction Broad phonetic classes (BPC) have widely been used in speech recognition research as, for instance, automatic language identification [1]; speaking rate estimation [2]; multi lingual systems [3,4] and, especially, in phone recognition [5-8]. Its success is due to the carried additional information that contributes to improve the recognition rates, especially under noise conditions [9]. In phone recognition task, BPC information may be used as an additional set of acoustic features or it may be integrated in the phone predictions. One successful example is presented by Siniscalchi et al. [6], one of the best results reported on the phone TIMIT recognition task, where 15 broad articulatory classes are used to predict posterior phone probabilities and to rescore phone lattices. Another interesting example is given by Morris and Fosler-Lussier [7] where the outputs of eight broad class classifiers are used as input features on a conditional random field (CRF) model. In [8], a hierarchical classification from successive broad classes is used with improvements in phone accuracy. Several other studies related to BPC could be referred, e.g. [5,10], sometimes using different terminology. In literature, BPC take different names (such as broad phonetic groups, speech attributes, events, etc.) but in all these studies they are always sets of phones with similar acoustic/phonetic features drawn manually by an expert (knowledge-driven information). The broad classes are selected according to acoustic-phonetic properties that derive from articulatory constraints or from hearing perception. This selection may contain some subjectivity or may be difficult to carry out when dealing with other kind of speech units like syllables or when considering coarticulation phenomena. BPC for the English language is an issue that has widely been addressed by the scientific community [5,7,8,11]. Its definition is usually related to the manner and place of articulation, so that all the broad classes show good agreement within some phonetic, articulatory and/or acoustic properties. The efficient construction of smaller/more compact phone sets is not trivial, however, as is confirmed by the lack of consensus between several proposals. For example in the TIMIT corpus, in [12], phone [dx] is classified as a sonorant consonant, while in [13] the same phone is classified as a stop and Halberstadt and Glass [10] classify it as a nasal/flap. The same happens with phone [hv]. In [12], it is a sonorant consonant and in [5,10] it is a fricative. Comparing the “fricative”broad classes of [5,14] it can be seen that the first proposal includes the phone [hv] while the second does not. The above examples are to show the subjective nature of the approaches based on human knowledge, even when referring to such a widely studied language as American English. * Correspondence: [email protected] 1 Instituto de Telecomunicações, Polo II, Coimbra P-3030-290, Portugal 2 ESTG, Instituto Politécnico de Leiria, Campus 2, Leiria P-2411-901, Portugal Full list of author information is available at the end of the article © 2012 Lopes and Perdigao; licensee Springer. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 http://asp.eurasipjournals.com/content/2012/1/158 This subjectivity affects not only the number of classes, but also the set of phones in each category. These problems, related to the expert-driven approaches, have prompted the emergence of data-driven approaches, where the classes’ composition is guided by data. We explore in this article a data-driven method where the broad classes are automatically defined according to the output of a speech recognition system. The method can be applied in systems using all kinds of recognition units (phones, syllables, subwords, phones of different languages, etc.) and do not conduct to a static division. The output of a classification system is usually evaluated comparing all the recognized sequences with the corresponding references. From this comparison, a confusion matrix can be computed. In the proposed approach it is considered that if a unit (phone or other) has much confusion with other unit is because, to the recognizer, they are somehow similar. This approach may not fully agree with phonetics principles or acoustic theories but it can be very helpful allowing to overcome deadlocks in some situations. Take the case of a multilingual system, where a knowledge-based approach has to involve experts from all the languages involved. The same occurs when dealing with varieties of a language (e.g., European Portuguese or Brazilian Portuguese). The proposed automatic clustering method extends the possibility of using broad classes’information to systems based on other than the phone unit. Data-driven clustering usually stands for a statistical measurement of a class. The key point of all clustering algorithms is the choice of a proximity or distance measure. This measure can be obtained from acoustic models, e.g., [3,15], or even rely on the confusion matrix, e.g., [4,16]. Model-driven methods and confusiondriven a methods are then the two major categories of datadriven phone clustering algorithms. In model-driven methods, the acoustic similarity between two phones can be achieved from the theoretical distance between the corresponding acoustic models. This distance can be the Bhattacharyya distance between two Gaussian mixture models [3,15]; a relative entropybased (Kullback–Leibler divergence) distance between two Laplacian mixtures [17], etc. In [18], a data-driven phonetic broad class generation is proposed where mutual information is used to compute similarity between models, while in [19] a similarity measure based on the likelihood between the acoustic frames and the hidden Markov models (HMMs) is proposed. In this article, we propose a confusion-driven method to generate phone clusters. This approach was already proposed in [4], where the phones are grouped using rules depending on a set of weights and thresholds. Our proposal differs from [4] in so far as we define a metric to compute phone distances. The remainder of the article is organized as follows. In Section 2, we introduce the data-driven approach and Section 3 presents the metric defined that evaluates the phones similarities. It also describes the way that the confusion matrix (which is the base for phone similarities computation) was achieved. Section 4 presents a system where broad classes are used in order to enhance phone recognition, and in Section 5 experimental results, comparing data-driven and knowledge-driven approaches, are presented. Finally, some conclusions and future improvements are drawn in Section 6. Data-driven broad classes In confusion-driven methods, the similarity measure approach comes from the phone confusion matrix M. This matrix is computed from the output of a phone recognizer by aligning the recognized sentence with the reference one, using a dynamic programming algorithm (the Levenshtein algorithm). It includes the concept of hits and confusions, as well as insertions (INS) and deletions (DEL). An example of part of a confusion matrix of our phone recognizer using TIMIT [20] data is shown in Figure 1. The diagonal refers to the number of correctly recognized phones (hits) and the off-diagonal elements refer to the number of misclassifications. The idea behind confusion-driven methods is that similar phones tend to be more confusable and should belong to the same class. Figure 2a shows, in pseudo colours, a confusion matrix using the 61 original TIMIT phones, where the phones are in alphabetical order and with blue representing the lower value and dark red the higher value. If highly confusable phones are repositioned in such a way that they become near each other, we get the matrix depicted in Figure 2b. In this second matrix, we can easily distinguish several sets (clusters) of phones where confusion between all the elements of the cluster is much higher than between other phones or phone clusters. In an attempt to find this set of clusters, this study explores RECOGNISED AS aa ae ah ao aw ax DEL SOURCE REFERENCE aa 125 ae 88 ah 111 ao 113 aw 14 ax 456 8 52 87 16 3 12 448 23 2 10 5 45 31 369 16 9 71 67 2 21 441 8 7 16 14 6 6 121 0 5 2 64 17 8 592 217 INS 21 21 19 17 14 30 Figure 1 Part of a confusion matrix: output of our TIMIT-based phone recognition system. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 2 of 12 http://asp.eurasipjournals.com/content/2012/1/158 an automatic classification method, where the phonetic classes are found according to the output of an automatic phone recognition system. Phones are grouped according to a similarity measure estimated from the confusion matrix given by the recognizer performance. Confusion-driven phone distance measure This section describes an automatic method for phonetic class generation based on a confusion matrix of a phone recognizer. Usually, the clustering techniques involve three concepts: 1. A data model; 2. A proximity criterion (similarity, distance, etc.); 3. A clustering algorithm that generates the clusters (broad classes) using the data model and the similarity measure. These concepts are discussed in following sections. Data model In the proposed method, the data comes from a confusion matrix yielded by the performance analysis of a phone recognition system. This matrix may contain results at the frame level, if artificial neural network (ANN), support vector machines (SVM), or CRF-based recognizers are used, or at the segment level, if segment models are used, such as HMMs or hybrid systems, as in the present case. An MLP/HMM hybrid system is used that combines an overall HMM structure with the class predictions given by a multilayer perceptron (MLP) classifier, thus benefiting from the time modelling abilities of HMMs and the discrimination capabilities of ANNs. In the TIMIT training set, this involves the recognition of 143 k segments. Hybrid MLP/HMM description An MLP network, with a single hidden layer, was trained for phone classification at a frame level. The last layer performs a 1-to-61 classification over the set of (TIMIT) phones. Speech was analysed every 10 ms with a 25-ms Hamming window. Thirty-nine parameters were used as standard input features representing 12 Mel Frequency Cepstral Coefficients (MFCCs), plus energy, and their first and second time-derivative coefficients. The context window used was 170 ms but only 9 frame features were used, as described in [8]. The current frame is in the centre of the context window (temporal information of past and future is included). The softmax function was used as the activation function of the output layer so that the output values could be interpreted as posterior probabilities. The hidden layer has 1,000 nodes and uses a sigmoid activation function. All the network weights and bias are adjusted using batch training with the resilient back-propagation (RPROP) algorithm [21], so as to reduce the minimumcross-entropy error between network output and the target values. The network has 413 k free training parameters. HMMs were built beforehand for each phone using HTK3.41, [22], in order to estimate the transition probabilities between states. Each phone was modelled by a three-state left-to-right HMM and each state was modelled by a single Gaussian model. The input features were the same as for the MLP. In the hybrid MLP/HMM system, the state likelihoods are replaced by the posterior probabilities given by the output predictions of the MLP. The three states share the same MLP output. We used HTK [22], with some changes in order to replace the usual Gaussian mixture models with the outputs of the MLP. The confusion matrix (data model) is computed using the entire TIMIT training set, which consists of all si and sx sentences of the original training set (3,698 utterances). The performance of the hybrid system is usually evaluated by means of Correctness (Corr) and Accuracy (Acc). We have used HTK evaluation tool HResults to Figure 2 (a) Confusion matrix of our TIMIT-based phone recognition system. (b) A repositioned version of the matrix depicted in (a) where highly confusable phones are place near each other. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 3 of 12 http://asp.eurasipjournals.com/content/2012/1/158 compute them. But with this tool only the token sequence is taken for evaluation. A fine evaluation takes into account not only the correct identified sequence of phones, but also their time localization. This was the criteria used in the confusion matrix generation on which the proposed clustering is based. A brief description of the evaluation procedure used is given below. A full description can be found in [23]. The evaluation procedure uses a modified Levenshtein algorithm [24] in the alignment between labelled and recognized phones, where the degree of overlapping between them is taken in the local distance definition. This algorithm finds the best alignment between two strings and inserts a penalty if an error occurs (insertion, deletion and substitution), but no penalty is applied if the labels match. In our proposal, we include an additional penalty that is proportional to the average of the left and right misalignments. If the labels do not overlap (T OV ≤ 0 in Figure 3), this penalty is set to a maximum value (p max ), such that an insertion or a deletion will be preferred to a misaligned substitution. Taking ti1;ti2and tj1;tj2as the boundaries of the test and reference labels, as indicated in Figure 3, they overlap if t i2 >t j1 or t j2 >t i1 and then the total time of the labels is T¼max ti2;tj2  min ti1;tj1  ¼T1þT2þTOV ð1Þ and the overlapping time is TOV ¼min ti2;tj2  max ti1;tj1  ð2Þ The left and right misalignments are T1¼tj1ti1  and T2¼tj2ti2 . If the labels lab i and lab j match their name but are not perfectly aligned, then we introduce an additional association penalty, p A (i,j), which is inversely proportional to the overlap between the labels, according to the following expression: pAi;jðÞ¼ T1þT2 ðÞ=2 TOV ¼1 2 T TOV 1  ð3Þ If the labels overlap more that 50%, p A is smaller than 0.5. As far as the overlapping decreases, this distance increases and is clipped to p max =15, which corresponds to 3.2% of overlapping. In order to promote confusions (substitutions) we penalize insertions and deletions significantly by setting p INS =12 and p DEL =12. The optimal alignment is found by tracing back the path of accumulated penalties from the last label pair to the origin of the (i,j) grid. With the alignment of all labels a confusion matrix can then be computed. This matrix represents the data model referred to in the beginning of Section 3. Similarity measure The confusion matrix is often converted into a symmetric similarity matrix using the so-called Houtgast algorithm. It measures the similarity between reference phones iand jusing the number of confusions of these phones with all other phones k. More specifically, if Nis the total number of classes (phones), the Houtgast similarity between phones iand j,sij is given by sij ¼sji ¼X N k¼1 min fik;fjk  ð4Þ with 1≤i;j≤N.Ifi=j,fij is the number of hits instead of confusions. According to this measure, two phones iand j are similar if they both have many confusions with the same phones and their similarity is zero only when phones iand jhave no simultaneous confusions with any phones. Figure 4 gives a simple example of this measure. The confusion matrix of a good phone recognizer is close to a diagonal matrix. The out-diagonal values represent misclassified phones (confusions between phones). Since the numbers of occurrences of each phone are quite different (due to phonetically unbalanced speech material) we normalize the confusion matrix by dividing the frequency counts f ij by the total number of occurrences of the phone iin the speech database: pij ¼fij P N n¼1 fin ¼P^ cjci jÞ ð5Þ In this way, we have an estimate of the probability of recognizing the model or cluster ^ cjwhen its reference class is c i . In this case the Houtgast similarity measure becomes s′ ij ¼X N n¼1 min pin;pjn  ¼X N n¼1 min P^ ci ðjcn ðÞ;P^ cj cnÞÞ ð6Þ T T 1 T 2 T OV Reference label, labj Test label, labi ti1tj1ti2tj2 Figure 3 Measurement of the time misalignment between two labels. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 4 of 12 http://asp.eurasipjournals.com/content/2012/1/158 This new measure has the following properties: (i) s′ ij≤1 and (ii) s′ ii ¼1 . Because min a;bðÞ¼ 1 2aþbab jj ðÞ and because PN n¼1pin ¼1 , a distance measure between phones iand jcan therefore be defined as d1ci;cj  ¼21s′ ij  ¼X N n¼1 pin pjn ð7Þ This distance forms a metric because it is the L 1 norm applied to row differences of matrix P (with elements p ij as defined in (5)). Several similarity measure proposals can be found in literature [4,16,25], but as referred in [26], they do not fulfil the properties of a proper metric. In the present proposal, d1has the three metric properties: it is positive, symmetric and satisfies the triangle inequality. Other distance measures can be defined based on the same principle, in particular the Euclidean distance of rows, d2ci;cj  ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi XN n¼1pin pjn  2 rð8Þ Clustering method The proposed clustering method groups phones in a multilevel hierarchy where clusters at one level are combined as clusters at the next level. Clustering can be achieved following hierarchical agglomerative clustering paradigm [27,28] as follows. Initially, each phone will be considered as a distinct cluster. Step 1 !Compute matrix P using (4). Step 2 !Find the distance between each pair of phones using (7). Step 3 !Compute the distances between all clusters. The distance between clusters r and s can be computed with several criteria, of which the simplest is the nearest neighbour:d r;sðÞ¼min dc ijr;cjs jÞÞ;i¼1...nr;  j¼ 1::nswhere n k is the number of phones in cluster k and c i|k is the ith phone in cluster k. Step4 !Create a new cluster by grouping the two nearest ones. Steps 3 and 4 will repeat until the number of desired clusters or a distance threshold is reached. Another possibility is to compute clusters until all the phones belong to the same cluster. A dendrogram can be built from these clusters, to allow the decision, the level or scale of clustering that is most appropriate for the application. TIMIT phone clustering results Starting from the data model described in Section 3.1 and following the steps proposed in Section 3.3, we arrive at a hierarchical phone clustering, depicted as a binary cluster tree (dendrogram) in Figure 5. The labels along the horizontal axis represent the phones in the original data set and vertical axis refers to the distance between the phones. This distance is computed from a similarity measure using d1ci;cj  . The number of clusters depends on where the tree is cut. In the example given in Figure 5, the 61 TIMIT phones are represented by the 9 clusters presented in Table 1. The consistence of the clusters, particularly the ones involving vowels, is well known. If we compare the resultant clusters with the knowledge-based division of [5] ab cde a b c d e a b c d e 10 2 1 3 0 a 16 4 3 4 6 111101 b4146 2 4 0 5 12 0 0 c 3 6 17 0 3 10 093 d4 2 0136 0 2 1 3 10 e 6 4 3 6 16 Confusion Matrix (M)Houtgast similarit y measure (S) 10 2130 111 1 0 1 1 + 2 + 1 + 0 + 0 = 4 Minimum between /a/ and /b/ pair of confusions Figure 4 Example of Houtgast similarity computation. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 5 of 12 http://asp.eurasipjournals.com/content/2012/1/158 presented in Table 2, only [y], [w] and [oy] are in a separate cluster. Nevertheless, these phones stay in a single cluster not because they are acoustically different but due to the little confusions with other phones. Another strong cluster is that of nasals. In this case, the datadriven division is the same as the knowledge-based division. Affricatives are set in a separate cluster (the knowledge-based proposal sometimes placed affricatives with stops and at others with fricatives). For fricatives and stops, the method relies on more than two clusters. The number of confusions between some fricatives and stops suggests that acoustically they exhibit similarities. The cophenetic correlation coefficient referred to in Figure 5 legend is a measure of how faithfully the dendrogram preserves the pairwise distances between the original unmodelled distances [29]. The closer the value of the cophenetic correlation coefficient is to 1, the more accurately the clustering solution reflects the data. Since the goal is phone recognition, the question remains as to how can these clusters help to improve phone recognition. The next section describes the followed approach—a hierarchical classification of different levels of phonetic information. Several intermediate classifiers offer posterior probability predictions for BPC, achieving phone detail at the end. Enhanced phone recognition In this section, we use the broad classes generated automatically by the method proposed in the previous section in a phone recognition task. Given the difficulty of finding a threshold that leads to an optimum number of clusters, we decided to cut the tree in different places, which resulted in several sets of clusters. This procedure forms a hierarchical structure, from broad to fine phonetic detail. The combination of several levels of phonetic detail has ih ix ax ah uh ae eh ax−h iy uw ux aa ao ow ay aw axr er r el l w oy b p d t dh v f th g kbcl dcl tcl pcl epi pau q g cl kcl hh hv ydx nx em en m n n g ch jh sh s z zh en g h# 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 ey Figure 5 Hierarchical phone clustering dendrogram using the average distance between clusters (cophenetic correlation coefficient = 0.873). Table 1 Sixty-one TIMIT confusion-division results in terms of nine clusters Clusters TIMIT-labelled phones Cluster 1 bcl dcl epi gcl kcl pau pcl q tcl Cluster 2 b d dh f g k p t th v Cluster 3 y Cluster 4 hh hv Cluster 5 dx em en m n ng nx Cluster 6 aa ae ah ao aw ax ax-h axr ay eh el er ey ih ix iy l ow oy r uh uw ux w Cluster 7 ch jh s sh z zh Cluster 8 eng Cluster 9 h# Table 2 Thirty-nine TIMIT knowledge-based division into five broad classes, from [5] Broad classes TIMIT-labelled phones Vowels l, r, w, y, er, ey, aw, ay, oy, ow, iy ,eh, ae, aa, uh, uw, ax, ix Stops p, t, k, b, d, g, jh, ch Fricatives s, z, zh, f, th, v, dh, hh Nasals m, n, ng Silences sil, dx Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 6 of 12 http://asp.eurasipjournals.com/content/2012/1/158 already been investigated in several studies, e.g. [5], where the outputs of four broad phonetic group classifiers (knowledge-based generated and trained separately) are combined in order to correct or enhance a phone classifier. Our proposal follows a similar approach, but by means of a hierarchical structure and with broad classes generated by the confusion-driven approach. Later, we show that this hierarchical classification takes advantage over the classifiers trained separately. Hierarchical broad classes We propose a hierarchical classification system that consists of a hybrid MLP/HMM, where the neural network architecture performs phone classification with a hierarchical set of broad class phonetic classifiers. The number of output layers in the MLP is the same as the cut places of the clustering tree, with the same order. Each cluster is characterized by the set of phones grouped by the tree cut and is called broad class. The broad class predictions from earlier classifiers are fed to the next ones in order to enhance the class discrimination in the current classifier. The last layer performs a 1-to-61 classification of the set of phones. All layers are trained concurrently so that, in training mode, targets are presented at all output layers. The serial arrangement proposed provides several broad-class posteriors along with the phone posteriors. A better phone classifier may be achieved if all these posteriors are correctly combined. Previous study [8] shows that phone prediction may be more robust if class membership probabilities are weighted and combined. A method for finding the best set of weights based on discriminative training in a hybrid MLP/HMM system is described below. Hierarchical MLP combination approach The goal of a combination approach is to take advantage of the broad-class posteriors along with the phone posteriors in order to improve the global phone recognition performance. Our approach considers that each phone can be predicted by combining all the broad-class outputs associated with that phone, with weights differing for each phone. These weights are found by means of a discriminative training method. Each weight will be assigned to the logarithm of the network output which includes the phone. The global phone posteriors are found by combining the corresponding outputs of all output layers. The proposed combination rule is expressed by ^ Pp kjyðÞ¼ 1 Zexp X NL l¼1 αlðÞ cklog ylðÞ ck  ! ð9Þ ^ Pp ky jÞð is the kth phone probability prediction, given the layer outputs, y, corresponding to the broad-class predictions in each of the N L output layers (see Figure 6). ylðÞ ckand αlðÞ ckare the network output and corresponding weight of layer land index c k , denoting the broad-class index (in layer l) to which the phone kbelongs. Each phone is predicted by weighting all the class outputs associated with the phone k,k21; ::; 61 fg , which are different for each phone. Referring to Figure 6, there are four output layers with 9, 16, 40 and 61 clusters and according to Table 3 the phone [zh] has indexes 3, 9, 23 and 61 in the layers from 1 to 4. In this equation, Zis a normalization factor for the predictor ^ Pp kyjÞðfor the 61 phones sum up to one. The best set of weights is the one which gives the highest phone accuracy according to our hybrid MLP/ Figure 6 Discriminative weight training method scheme. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 7 of 12 http://asp.eurasipjournals.com/content/2012/1/158 HMM recognition system. An iterative training method based on the paradigm of discriminative training is therefore appropriate. Discriminative training of the weights Every kind of error should be considered (substitutions, insertions and deletions) in the definition of a cost function. Since these errors are found by the Levenshtein distance, the objective function should include a minimization of this distance with multiple hypotheses. However, we used a simple 1-best discriminative function, thereby avoiding the error counting, which it is better than applying the phone targets to the network output layer as is usually done. The Levenshtein distance aligns two label sequences. One is the reference (assumed correct) sequence, Wlab , and the other is the best decoding hypothesis given by the recognizer, Wrec . Using the Viterbi algorithm, we define an error function as dW rec;Wlab  ¼gW rec ðÞgW lab ðÞ ð10Þ where gW lab ðÞand gW rec ðÞrepresent the reference and best acoustic log likelihood of the observation sequence according to the Viterbi algorithm. This difference corresponds to a likelihood ratio. It is always greater than zero and is only zero if the two transcriptions are exactly the same (if labels and time alignments match). If N BD is the total number of training utterances, the global cost is then given by E¼X NBD n¼1 dW nðÞ rec ;WnðÞ lab  ð11Þ In the hybrid MLP/HMM approach, the a priori probability function b s (x) is the likelihood of observing x in the HMM state, s, being transformed in the posterior probability predicted by Equation (9). In order to find the appropriate set of weights αlðÞ k no , a gradient descendent method is applied. In this case, it can be shown that the error gradient has terms of the form @ @αlðÞ ck log^ Pp ky jÞ¼ log ylðÞ ck  11=ZðÞ ð12Þ On the other hand, the gradient of the weights is rαlðÞ ck¼@E @αlðÞ ck ¼X NBD n¼1 @gW nðÞ rec  @αlðÞ ck  @gW nðÞ lab  @αlðÞ ck 0 @1 Að13Þ The gradients of the log likelihoods in this expression depends on (12) because the HMM states of the path (WnðÞ rec or WnðÞ lab ) are associated with the MLP outputs, ylðÞ ck. Figure 6 illustrates the scheme of the proposed discriminative weight training method. Table 3 Description of data-driven broad classes BPC9 BPC16 BPC40 l, el, w, r, er, axr, ey, aw, ay, iy, ih, eh, ae, ah, ax, uh, ix, uw, ux, ax-h, aa, ao, ow, oy l, el, w l, el w r, er, axr r, er, axr ey, aw, ay, iy, ih, eh, ae, ah, ax, uh, ix, uw, ux, ax-h, aa, ao, ow ey aw ay iy ih, eh, ae, ah, ax, uh, ix uw, ux ax-h aa, ao, ow oy oy p, b, d, t, f, th, v, dh, k, g p, b p b d, t, f, th, v, dh d, t f,th v, dh k, g k g jh, ch, sh, z, s, zh jh, ch, sh jh, ch sh z, s, zh z, s zh hh, hv hh, hv hh, hv m, n, em, ng, en, nx, dx m, n, em, ng, en m , n em ng en nx, dx nx dx pcl, tcl, dcl, kcl, q pcl, tcl, dcl, kcl, q pcl bcl, gcl, pau, epi bcl, gcl, pau, epi tcl, dcl kcl q bcl gcl pau, epi yyy eng eng eng h# h# h# Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 8 of 12 http://asp.eurasipjournals.com/content/2012/1/158 We used the RPROP to accelerate the convergence to a solution. Results of the proposed hierarchical MLP architecture with broad classes found by the clustering method are presented in Section 5.3. Experimental results All the experiments were carried out using hybrid MLP/ HMM systems. Speech was analyzed as in Section 3.1.1. Both training and testing were carried out using the TIMIT database [20] and its original 61 phoneme set. This database is often used in phone recognition benchmarking, e.g. [5,7,30]. The train and test sets correspond to the original splitting of the TIMIT database. While the training set with all si and sx sentences has 3,698 utterances, the test set consists of all si and sx sentences from the complete 168-speaker test set, which has 1,344 utterances. The targets derive from the phone boundaries provided by the TIMIT database. Although the neural network is tailored to discriminate the full 61 TIMIT phones, these symbols are sometimes considered a too narrow description for practical use, and for evaluation purposes we collapsed the 61 TIMIT labels into the standard 39 phones as proposedbyLeeandHon[31].InthehybridMLP/HMM systems, the a priori state likelihoods are replaced by posterior probabilities, ^ Pp kx jÞð , given directly from the MLP output layer or according to Equation (9) (Section 4.2). The performance of the MLPs was evaluated by means of a frame error rate (FER). The performance of the hybrid system was evaluated by means of Correctness (Corr) and Accuracy (Acc). Hierarchical versus single layer classification As described in Section 4.1, a hierarchical classification of different levels of phonetic information is proposed in order to improve phone recognition. The neural network has about 161 k parameters. It comprises eight hidden layers. The number of nodes in the layers is (in numerical order): 50-9-50-16-50-40-100-61. Even layers give posterior probability predictions for BPC and the last layer gives posterior probability predictions for phones. Thus, the proposed MLP system is trained as a function of the 61 phones and 3 additional sets of BPC, consisting of 9, 16, 40 TIMIT phone sets (viz. BPC9, BPC16, BPC40) achieved by the confusion-driven clustering approach proposed. The hierarchical phone clustering dendrogram in Figure 5 gives rise to the BPC9. The two others BPCs result from cutting the data-driven dendrogram tree into two other levels. The resulting sets of BPCs were grouped according to the division presented in Table 3. Standard MFCC’s features and derivatives are presented for input (odd) hidden layers. A context window of 290 ms was used using only 15 frame features (details in [8]). All layers were trained concurrently so that, in training mode, targets were presented at all even layers: layers 2, 4, 6 and 8. Since even layers are trained with a softmax activation function, its outputs can be seen as BPC probabilities. The odd uses a sigmoid activation function. All the network weights and bias are adjusted using batch training with an RPROP algorithm [21] so as to minimize the minimum-cross-entropy error between the network output and the target values. The choice of the error function followed Bishop’s suggestion [32], which was later clarified by Dunne and Campbell [33]. It states that the softmax activation function should couple with the cross-entropy penalty function. In order to evaluate the performance of the proposed hierarchical network, we trained four single layer networks, each one specialized in one BPC. Each single layer network was trained with the same number of hidden nodes as the hierarchical network (e.g. BPC16 MLP has 50 hidden nodes and an output layer with 16 outputs). The results in terms of FER are given in Figure 7 and show that the hierarchical network outperformed the equivalent single layer networks with respect to BPC16, BPC40 and 61 TIMIT phones. Regarding the classification of BPC9 (9 clusters of Table 1) the performance of the single layer network is similar to the hierarchical. These results encourage the use of the hierarchical structure in further experiments. Note that these results relate to an evaluation (in terms of FER) of the test set for every training epoch. Hierarchical confusion-driven versus hierarchical knowledge-driven phone recognition Since confusion-driven clustering is the topic of this article, the performance of this kind of clustering has to be compared with that achieved by an expert-knowledge approach. Therefore, we trained two hierarchical MLPs from 0 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 Train Epochs FER FER 9 classes-hierarchical net FER 16 classes-hierarchical net FER 40 classes-hierarchical net FER 61 classes-hierarchical net FER 9 classes-single layer FER 16 classes-single layer FER 40 classes-single layer FER 61 classes-single layer Figure 7 FER comparison of hierarchical BPC’s classification and equivalent single layer BPC’s classification. Lopes and Perdigão EURASIP Journal on Advances in Signal Processing 2012, 2012:158 Page 9 of 12 http://asp.eurasipjournals.com/content/2012/1/158