Improving Spanish Polarity Classification Combining Different Linguistic Resources

Improving Spanish Polarity Classification Combining Different Linguistic Resources Eugenio Mart´ınez-C´amara1(B),Ferm´ın L. Cruz2, M. Dolores Molina-Gonz´alez1, M. Teresa Mart´ın-Valdivia1, F. Javier Ortega2, and L. Alfonso Ure˜na-L´opez1 1SINAI Research Group, University of Ja´en, Campus Las Lagunillas, 23071 Ja´en, Spain [email protected] 2Department of Languages and Computer Systems, University of Seville, Av. Reina Mercedes s/n, 41012 Sevilla, Spain Abstract. Sentiment analysis is a challenging task which is attracting the attention of researchers. However, most of work is only focused on English documents, perhaps due to the lack of linguistic resources for other languages. In this paper, we present several Spanish opinion mining resources in order to develop a polarity classification system. In addition, we propose the combination of different features extracted from each resource in order to train a classifier over two different opinion corpora. We prove that the integration of knowledge from several resources can improve the final Spanish polarity classification system. The good results encourage us to continue developing sentiment resources for Spanish, and studying the combination of features extracted from different resources. Keywords: Sentiment analysis ·Polarity classification ·Lexicon-based approach ·Sentiment feature generation 1 Introduction Sentiment classification or polarity detection is an opinion mining task oriented to determine the overall sentiment-orientation of the opinions contained within of a given document. The document is supposed to contain subjective information such as product reviews or opinionated posts in blogs. This task has been widely studied, but most of the research is focused on dealing with English documents, perhaps due to the lack of resources in other languages. However, opinions and comments in the Internet are expressed using other languages different from English such as Chinese, Spanish or Arabic. The development of This work has been partially supported by a grant from the Fondo Europeo de Desarrollo Regional (FEDER), ATTOS project (TIN2012-38536-C03-0) from the Spanish Government. The project AORESCU (P11-TIC-7684 MO) from the regional government of Junta de Andaluc´ıa partially supports this manuscript, and the project CEATIC-2013-01 from the University of Ja´en. new linguistic resources is essential to make progress in solving the problem of the sentiment analysis, being even higher in languages other than English, like Spanish. Therefore, it is important to develop resources to help researchers to work with these languages. In this paper we combine several Spanish resources for opinion mining in order to improve the final system for polarity classification. Specifically, we are going to combine two Spanish lexicons that have been developed in very different ways. On the one hand, iSOL (improved Spanish Opinion Lexicon) resource [21] has been derived from the English Lexicon of Bing Liu [14]and then it has been manually revised and improved. On the other hand, the ML-SentiCon [6,8] (Multi-Layered, Multilingual Sentiment Lexicon) resource is based on an improved version of SentiWordNet [1]. Several features were calculated using iSOL and ML-SentiCon to represent each document. Also, the combination of the features calculated using the two lexicons has been study. Support Vector Machine was the algorithm selected to analyse the goodness of the different sets of features. Our main goal is to demonstrate that the combination of both different resources can improve the final result. In order to prove the robustness of the method, we have carried out experiments over two different corpora of reviews: a corpus of movie reviews called MuchoCine [5] and a corpus of opinions about hotels called COAH (Corpus of Opinions from Andalusian Hotels) [22]. Both corpora include comments and reviews written in Spanish. However, they have meaningful differences not only related to the domain tackled but also according to the number of reviews, comment size, balance between positive and negative samples, and even the nature of the information contained, more descriptive in movies and more subjective in hotels. In addition, the experiments also reinforce the idea already depicted in other works that the movie domain is more difficult to learn than hotel domain [3]. Therefore, the polarity in sentiment analysis not only remains through different languages [20], but also we have proved that the difficulty of a domain is also preserved across the languages. The paper is organized as follows: Next section includes a review of some works related to polarity classification using other languages than English and specifically dealing with Spanish. Section 3presents the resources used in our experiments paying more attention to the corpus COAH and the lexicon MLSentiCon. Experimental framework is described in Sect. 4and the analysis of results is presented in Sect. 5. Finally, main conclusions and further work are expounded in Sect. 6. 2 Related Works Polarity classification has been mainly faced from two points of view: machine learning techniques based approaches and approaches based on the semantic orientation of words. The first group is wider used for the classification of reviews. In this type of approaches, the document is represented by different features that may include the use of n-grams or defined grammatical roles like adjectives or other linguistic feature combinations. Then a machine learning algorithm is applied, usually Support Vector Machine (SVM), Maximum Entropy (ME) and Nave Bayes (NB). A survey of studies using machine learning on this task can be found in [17,23]. On the other hand, there is a lot of work based on the semantic orientation approach, which represent the document as a collection of words, computing the polarity of the document as an aggregation of the polarity of its words. The sentiment of each word can be determined by different methods, for example using a web search [12] or consulting a lexical database like WordNet [15]. Among the methods that consider some linguistic features in order to determine the sentiment at a word-level we can highlight many studies in the literature [9,11, 27,30]. Part of our proposal is based on the work [14]. Regarding polarity classification on non-English languages, there are a number of interesting studies that apply a semantic orientation approach based on sentiment words. Many of them use resources on English and then apply some translation process in order to apply the English-based resources to the target language. Kim and Hovy [16] compare opinion expressions within an aligned corpus of emails in German and English. They translate English opinion-bearing words into German and then analyze German emails using those translated words. Zhang et al. [31] apply sentiment analysis to two Chinese datasets: the first one contains opinions on euthanasia from different web sites, while the second dataset included Chinese reviews about products in six categories from Amazon. They propose a rule-based approach including two phases: firstly, determining each sentences sentiment based on word dependency, and secondly, aggregating those values of sentiment at a sentence level in order to predict the document sentiment. Wan [29] studied how to reduce the need of using linguistic resources for sentiment analysis for texts in Chinese. The author followed a supervised approach and proposed a co-training system based on the use of an English corpus for polarity classification of Chinese products reviews applying a machine translation system. There are also some remarkable studies using SO based on bearing-words lists. For example, Banea et al. [2] proposed several approaches to cross-lingual subjectivity analysis by directly applying the translations of opinion corpus in English to the training of an opinion classifier in Romanian and Spanish. This work showed that automatic translation is a viable alternative for the construction of resources and tools for subjectivity analysis in a new target language. Cruz et al. [5] gathered a corpus of Spanish movie reviews from the MuchoCine website. MuchoCine (MC) corpus was manually annotated and used for the development of a polarity classifier based on the semantic orientation of the words. On the other hand, Brooke et al. [3] presented several experiments dealing with Spanish and English resources. They conclude that, although the machine learning techniques can provide a good baseline performance, it is necessary to integrate language-specific knowledge and resources in order to achieve a noticeable improvement. They proposed three approaches: the first one uses manually and automatically generated resources for Spanish. The second one applies machine learning to a Spanish corpus. The last one translates the Spanish corpus into English and then applies the SO-CAL, (Semantic Orientation CALculator), a tool developed by themselves [26]. Finally, it is worth to mention the work in [24], where a framework for the generation of sentiment lexicons in a target language is presented. They use manually and automatically annotated English resources and then map these annotations to other languages by using the multilingual aligned WordNet family. Using this method, they build two Spanish lexicons with 1,347 and 2,496 terms, respectively. It is interesting to highlight the evaluations performed through the manual annotation of a test set of 100 terms from each lexicon, achieving an accuracy of 90 and 74 %, respectively. 3 Resources In this paper we will use four Spanish resources for opinion mining. Firstly, a Spanish corpus of hotel reviews has been compiled. This corpus, called COAH (Corpus of Opinion about Andalusian Hotels) [22], is a new resource for Spanish community in opinion mining. Secondly ML-SentiCon [6,8], a new resource composed by lemma-level sentiment lexicons for English, Spanish and other three official languages in Spain (Catalan, Basque and Galician). Finally, the other corpus and lexicon used for our experiments are the Spanish corpus of movie reviews called MuchoCine Corpus [5] and the domain-independent lexicon iSOL [21] already presented in other works. We briefly introduce these resources in next subsections. 3.1 Lexicons The iSOL resource was generated from the Bing Liu English Lexicon [14]by automatically translating it into Spanish and obtaining the SOL (Spanish Opinion Lexicon) resource. Then this resource was manually reviewed in order to improve the final list of words obtaining iSOL (improved SOL). The iSOL is composed of 2,509 positive and 5,626 negative words, thus the Spanish lexicon has 8,135 opinion words in total. This resource has been successfully evaluated in [21] using the MuchoCine corpus. The results showed that the use of an improved list of sentiment words from the same language could be considered a good strategy for unsupervised polarity classification. On the other hand, ML-SentiCon is a set of lemma-level sentiment lexicons for English, Spanish and other three official languages in Spain (Catalan, Basque and Galician). The lexicons are induced using an automatic, semi-supervised method and are formed by 8 layers, allowing applications to choose different compromises between the amount of available words and the accuracy of the estimations of their prior polarities. For each POS tagged lemma in the resource, they are provided two scores: a real value representing the prior polarity, between −1and1, and the standard deviation reflecting the ambiguity of that value. According to manual verification of a significant sample, the lexicons for English and Spanish have both high accuracies, over 90 % for layers 1–6 and 1–5, respectively Table 1. Sizes and accuracies of English and Spanish ML-SentiCon lemma-level lexicons Layer English Spanish #Lemmas Accuracy #Lemmas Accuracy 1 157 99.36 % 353 97.73 % 2 982 98.88 % 642 97.20 % 31,600 97.75 % 891 94.95 % 42,258 96.24 % 1,138 93.06 % 53,595 93.95 % 1,779 91.75 % 66,177 91.99 % 2,849 86.09 % 713,517 85.29 % 6,625 77.69 % 825,690 74.06 % 11,918 61.29 % (Table 1). In the case of the Spanish lexicon, the accuracy is sensibly better than the accuracy reported in other recent work [24]. The lemma-level lexicons were automatically created from a synset-level lexicon for English, which in turn were built with an enhanced version of the method used by Baccianella et al. [1] to build SentiWordNet 3.0, one of the most used sentiment lexicons nowadays. This method comprises two steps, one involving the classification of individual synsets from WordNet as positive, negative or neutral, and another one involving a global, graph-based quality improvement of the positivity and negativity scores of the synsets. Several improvements were added in both steps. In the first one, a new source of information was used for training the classifiers, WordNet-Affect 1.1 [25] and a meta-learning scheme for combining multiple classifiers was applied. In the second step of the method, they were proposed new kinds of WordNet-based graphs, including positive and negative arcs, and a different random-walk algorithm called PolarityRank [7]. Evaluations of the positivity and negativity scores obtained in each step show significant improvements with respect to the original method. The lemma-level English lexicon was built by computing the means of positivity and negativity scores from those synsets corresponding to each POS tagged lemma. The lemmas were distributed over layers by gradually relaxing a set of restrictions (Table 1). In this way, the number of lemmas that satisfy the restrictions increases in each layer, at the same time as the reliability of those lemmas as indicators of positivity and negativity decreases. The Multilingual Central Repository 3.0 [10] and some resources from the EuroWordNet project [28] up to November 2009 were used in order to link synsets to lemmas for other languages in ML-SentiCon: Spanish, Catalan, Basque and Galician. 3.2 Corpora In this paper we have used two different corpora. MuchoCine has been already described in [5] and it has widely used in several works [13,18,19,21]. The corpus consists of 3,878 movie reviews collected from the MuchoCine website. The reviews are written by web users instead of professional film critics. This increases the difficulty of the task because the sentences found in the documents may not always be grammatically correct, or they may include spelling mistakes or informal expressions. The corpus contains about 2 million words and an average of 546 words per review. On the other hand, we have collected the Corpus of Opinion about Andalusian Hotels COAH, from the TripAdvisor site. The collection contains 1,816 reviews which were written by non-professional reviewers, rather than web users. Similarly to what happens with movie reviews, the texts in hotel reviews may not be grammatically correct, or they can include spelling mistakes or informal expressions. We have selected only Andalusian Hotels: per each province of Andalusia (Almera, Cdiz, Crdoba, Granada, Jan, Huelva, Mlaga and Sevilla) we have selected ten hotels, five of them with higher ratings and the other five with worse ratings. All the hotels must have at least twenty opinions in the latter years written in Spanish. As a result of these constraints, we have obtained 1,816 reviews. Finally, the corpus contains reviews for 80 hotels with an average of 23 reviews per hotel. We want to highlight that the hotel reviews are composed by about 145 words with a mean close to ten adjectives. For both corpora the opinions are rated on a scale from 1 to 5. A rank of 1 means that the opinion is very bad, and 5 means very good. Reviews with a rating of 3 can be categorized as neutral which means the user consider the hotel/movie is neither bad nor good. Table 2shows the number of reviews per rating for each corpus. Finally, in our experiments the neutral reviews were discarded. In this way, opinions rated with 3 were not considered, the opinions with ratings of 1 or 2 were considered as positive and those with ratings of 4 or 5 were considered as negative. A total of 2,625 reviews were processed for MuchoCine Corpus (1,274 positives and 1,351 negatives) and 1,816 comments about hotels were considered for COAH corpus (1,020 positives and 511 negatives). Table 2. Rating distribution for MuchoCine and COAH corpora Rating #Reviews in MC Reviews in COAH 1 351 312 2 923 199 31,253 285 4 890 489 5 461 531 Total 3,875 1,816 4 Experimental Frameworks As it has been mentioned above, two are the main contributions of this research. The first one is focused on the comparison of two linguistic resources for opinion mining in Spanish: iSOL and ML-SentiCon. Thus, two polarity classification systems were developed and they were assessed over two opinion mining corpora in Spanish. The fact that the two corpora are centred in two different domains must also be highlighted. One of the main issues of a text classification system is the selection of a good set of features. The selection of the features in a sentiment classification system is critical, because those features have to represent the polarity or the intention of the author. The two lexicons herein compared offer different kind of sentiment information, so the second bunch of experiments are focused in the combination of the features calculated with iSOL and with ML-SentiCon with the aim of improving the polarity classification in Spanish. The combination of features from two different linguistic sources constitutes the second main contribution of this article. 4.1 Individual Experiments Due to the different nature of the linguistic resources, two polarity classification systems have been developed. Concerning iSOL, two polarity classification systems have been evaluated. Each review is represented as a vector of features that are computed using the lexicon iSOL. The two corpora have two sections where the authors briefly express their overall opinions (a title or summary of the review), and the complete opinion about the movie or the hotel (body of the review). The summary is not necessarily an excerpt of the body. The main characteristic of the summary is that it is more concise and the author expresses the opinion more clearly. This is the main reason of treating independently the two sections of the documents. So, the system extracts features from the two sections separately. Thus, per each document four features are calculated: 1. Number of positive words in each part of the document (two features). 2. Number of negative words in each part of the document (two features). After calculating the features of the documents, a 10-fold cross-validation evaluation is carried out with the goal of assessing the goodness of the bunch of features. The machine learning algorithm selected was SVM1, using a linear kernel and normalizing the feature vectors. The results obtained are shown in Table 3. Regarding ML-SentiCon, a supervised polarity classification system has been developed. Each review, taken from each of the two corpora, is represented as a vector of 48 features. The features are calculated taking the information of each of the layers of the ML-SentiCon. As this resource is composed of POS-tagged 1The SVM implementation was LibSVM [4]. Table 3. Results obtained with iSOL lexicon Corpus Accuracy Precision Recall F1 MC 66.01 % 66.12 % 66.01 % 66.02 % COAH 92.09 % 92.21 % 92.09 % 91.94 % Table 4. Results obtained with ML-SentiCon lexicon Corpus Accuracy Precision Recall F1 MC 65.37 % 65.47 % 65.37 % 65.37 % COAH 89.09 % 89.07 % 89.09 % 88.88 % lemmas, each review needs to be lemmatized and POS-tagged. MuchoCine corpus already contains this information. In the case of COAH corpus, we applied the same analysis tool used in the MuchoCine corpus, FreeLing (Padr and Stanilovsky, 2012). Per each layer and section of the documents three features are calculated: 1. Sum of polarities of all lemmas from that layer appearing in that section. 2. Sum of polarities of positive lemmas from that layer appearing in that section. 3. Sum of polarities of negative lemmas from that layer appearing in that section (Table 4). Once the documents are represented as vectors of features, a 10-fold crossvalidation evaluation is applied with the aim of assessing the goodness of the set of features. The machine learning algorithm selected was the same as the experiments realised with iSOL, i.e. SVM, using the same exact parameters. The results obtained over the two corpora are shown in Table 5. 4.2 Combining Lexicons The similar results obtained with the two sets of features encourage us to combine the features generated with each lexicon. Both resources have been developed with quite different approaches, which suggest they provide different information to a certain extent. Since each document in ML-SentiCon is represented with 48 features and documents in iSOL have 4 features, in the union of the two sets of features each document is represented with 52 features. The same evaluation process has been repeated, i.e., 10-fold cross-validation evaluation and SVM as machine learning algorithm. The results reached are shown in Table 5. 5 Analysis of Results In the analysis of the results three points could be highlighted: the differences between the two domains (movies, hotels), the differences between the two linguistic resources, and the improvement reached with the combination of the two sets of features. Table 5. Results of the lexicon features union Corpus Accuracy Precision Recall F1 MC 68.38 % 68.41 % 68.38 % 68.38 % COAH 93.79 % 93.78 % 93.79 % 93.74 % It is evident that there is an important difference between the results obtained with the two domains. The first conclusion is that it is easier to classify opinions in reviews of hotels than in reviews of movies. This assert is confirmed when the documents of each corpus are analysed. The main difference is the length of the reviews: meanwhile the movie reviews are long, hotel reviews are concise. The greater length of the movie reviews indicates that the authors not only express their feelings about the movie, but also wrote a synopsis. The authors use in the description of the movies polar words that are in the lexicons, but they are not been used as a part of an opinion of the author. Furthermore, the systems developed to evaluate the lexicons do not distinguish whether a sentence is subjective or objective, the systems are only polarity classifiers. Thus, the systems developed tend to misclassify the movie reviews where the author also summarizes the plot of the movie. On the other hand, the hotel reviews are more succinct, and the authors express their opinion on the hotel and not a description of the accommodation. Thus, when the authors use a polar word is more likely to be part of an opinion. This conclusion about the different difficulties learning a specific domain has been already depicted in other works, however this is the first time that has been proved for Spanish. It is very clear that the two linguistic resources are totally different, while iSOL is a list of positive and negative words, ML-SentiCon is a ranking of words layered. The results reached with the two sentiment lexicons are good, but we have to say that iSOL reached slightly better results. This is because iSOL is a sentiment lexicon compiled semi-automatically and revised manually, while ML-SentiCon has been compiled fully automatically and has not been manually corrected. An interesting result is reached when the features calculated with the two sets of resources are joined. In both domains the results are improved, so it proves that the information obtained from the two lexicons is complementary. Thus, it is not fair to say that a resource is better than the other because each of them obtains different information that allows performing a good polarity classification in Spanish, but when they are combined the classification is even better than when the features are classified separately. 6 Conclusions The main contributions of this paper are the use of different Spanish resources for opinion mining. Specifically, the new sentiment lexicon for Spanish, MLSentiCon and the new corpus called COAH, have been firstly used for Spanish