Software Repository Mining Analytics to Estimate Software Component Reliability

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Software Repository Mining Analytics to Estimate Software Component Reliability André Freitas - freitas.andr[email protected] Mestrado Integrado em Engenharia Informática e Computação Supervisor: Rui Maranhão - [email protected] Co-Supervisor: Alexandre Perez - [email protected] July 26, 2015 Software Repository Mining Analytics to Estimate Software Component Reliability André Freitas - freitas.andr[email protected] Mestrado Integrado em Engenharia Informática e Computação Approved in oral examination by the committee: Chair: Doctor Hugo José Sereno Lopes Ferreira External Examiner: Doctor Jâcome Miguel Costa da Cunha Supervisor: Doctor Rui Filipe Maranhão de Abreu July 26, 2015 Abstract Finding and fixing software bugs is expensive and has a significant impact in Software development effort. Repositories have hidden predictive information about Software history that can be explored using analytics and machine learning techniques. A Software component can be a file, class or method in terms of granularity. Current research in Mining Software Repositories (MSR) is capable of ranking and listing faulty components at the file granularity. Crowbar is an automatic Software debugging tool that uses a technique named Barinel. Our goals are predicting Software defects with method granularity and improve Crowbar, by mining repositories. We have implemented a tool named Schwa, available for free on Github, that is capable of analyzing Git repositories. We are analyzing metrics such as revisions, fixes, authors and the time of commits to feed the prediction model. The analysis of time provides a method to ignore old components. Experimental results shown that for every Software repository, the predictive power of each metric is different. For example, in some projects revisions is more correlated with future defects and in others is fixes. The usage of defect predictions from Schwa in Crowbar reduced the amount of time necessary to rank faulty components. In the Joda Time project the time was reduced from one hour to less than a minute. This thesis does the following contributions: a method to parse and represent diffs from patches with method granularity for Java; a model to compute defect probabilities; a framework for mining Software repositories; a technique to learn the importance of tracked metrics; a method to evaluate the gain of using defect probabilities in fault localization. i ii Resumo Encontrar e corrigir bugs tem um grande custo e impacto no esforço em desenvolver Software. Os repositórios escondem informação preditiva sobre o histórico de Software que pode ser explorada recorrendo a técnicas de análise e de machine learning. Um componente de Software pode ser um ficheiro, classe ou método em termos de granularidade. A investigação atual de Mining Software Repositories (MSR) é capaz de classificar e listar componentes defeituosos com a granularidade ao nível do ficheiro. O Crowbar é uma ferramenta que faz depuração automática de Software e usa a técnica Barinel. Os nossos objetivos são prever defeitos em Software com granularidade até ao método e melhorar o Crowbar, ao extrair informação de repositórios. Foi implementada uma ferramenta denominada de Schwa, disponível livremente no Github, que é capaz de analisar repositórios Git. Estamos a analisar métricas como as revisões, correções de bug, autores e o tempo dos commits para alimentar o modelo de previsão. A análise do tempo permite ignorar componentes mais antigos. Os resultados experimentais demonstraram que para cada repositório de Software, o poder preditivo de cada métrica é diferente. Por exemplo, em alguns projetos o número de revisões está mais correlacionado com futuros defeitos e em outros é o número de correções de bugs. A utilização das previsões de defeito do Schwa no Crowbar reduziu o tempo necessário para classificar componentes faltosos. No projecto Joda Time o tempo foi reduzido de uma hora para menos de um minuto. Esta tese faz as seguintes contribuições: um método para interpretar e representar diffs de patches com a granularidade ao método; um modelo para calcular probabilidades de defeito; uma framework para minar repositórios de Software; uma técnica para aprender a importância das métricas analisadas; um método para avaliar o ganho de usar as probabilidade de defeito em localização de falhas. iii iv Acknowledgements First, I would like to thank my supervisor and co-supervisor, Rui Maranhão and Alexandre Perez for their extraordinary help and mentoring in my dissertation, specially for supporting me on the most difficult challenges. Thanks for accepting me as a dissertation student and for your patience through the last months. I would like to thank my supervisor for the financial aid I received through FCT funding, since it was an important help for me. I would like to thank Nuno Cardoso for helping me through the internals of Crowbar. Regarding my experiments, I would like to thank the contributions from Shiftforward, Luís Fonseca, Diogo Pinela and Stronsgtep. Thanks Open Source community for making available software for free, that students and researchers frequently use on their projects. Thanks FEUP for having a good environment, teachers and for everyone that indirectly contributed to the success of this thesis that I could not list here. Finally and not least important, I would like to thank my parents, my family and my girlfriend for always supporting me. They surely gave me an environment to be a better person and pursuing my goals. André Freitas v LIST OF FIGURES xii List of Tables 2.1 Repository example - files and commits . . . . . . . . . . . . . . . . . . . . . . 10 2.2 Repository example - commits . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Programspectrum.................................. 13 3.1 Examplesofranks.................................. 28 4.1 Schwaresults .................................... 32 4.2 Libcrowbarresults ................................. 32 4.3 JodaTimeresults .................................. 33 4.4 MeoArenaresults.................................. 33 4.5 MongoJavadriverresults.............................. 33 4.6 Scraimresults.................................... 34 4.7 Trainsimresults................................... 34 4.8 Adstaxresults.................................... 34 4.9 Boxerresults .................................... 35 4.10Apsoresults..................................... 35 4.11Hivedbresults.................................... 35 4.12Teamengineresults ................................. 35 4.13Automatalibresults................................. 36 4.14CDITCKresults .................................. 36 4.15 Commits applied to Joda Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.16 Schwa configurations for Joda Time . . . . . . . . . . . . . . . . . . . . . . . . 38 4.17 Diagnostic cost for Joda Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.18 Commits applied to CDI TCK . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.19 Schwa configurations for CDI TCK . . . . . . . . . . . . . . . . . . . . . . . . 40 4.20 Diagnostic cost for CDI TCK . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 xiii LIST OF TABLES xiv Abbreviations MSR Mining Software Repositories TWR Time-Weighted Risk SCM Source Control Management SFL Spectrum-based Fault Localization MDB Model Based Diagnostic ISTQB International Software Testing Qualifications Board SaaS Software as a Service LRU Least Recented Used URL Uniform Resource Locator PHP PHP: Hypertext Preprocessor CSS Cascading Style Sheets HTML HyperText Markup Language MIT Massachusetts Institute of Technology xv Chapter 1 Introduction We review the state of the art in Mining Software Repositories (MSR), existing tools and propose a new method to predict defects based on data extracted from repositories. Also we use this information from defect prediction to improve the diagnostic accuracy from Crowbar, namely, the Barinel algorithm. 1.1 Context Software plays an important role for society and in our daily routine, since we use applications to communicate, manage information, etc. We expect that these applications behave correctly and we are easily frustrated when they are defective. Development of software is not a simple task since developers need to maintain complex code, test and manage expectations of stakeholders by correctly interpreting requirements. It is estimated that fixing bugs represent 90% of development costs [Ser13]. There are tools that can help developers delivering high quality software, by automatically reviewing code and analyzing their behaviour. Some of these tools are Codacy1, Crowbar2and Codeclimate3. The usage of revision control systems such as Git, SVN and Mercurial, helps developers tracking changes on Software and understanding the evolution of components. Tools are important to developers since they automate and avoid repetitive tasks in Software development. With the growing usage of revision control systems, research in MSR evolved in the last decade and involves the analysis of systems used to support the development of software such as repositories, issue trackers and mailing lists [HNB+13]. 1https://codacy.com/ 2http://crowbar.io/ 3https://codeclimate.com/ 1 Introduction 1.2 Motivation and goals Software repositories have hidden information that can be explored with analytics and machine learning techniques to support defect prediction models. The Barinel algorithm in Crowbar uses static estimations for defect prediction and goodness of components [AZG09]. Insights from software history could substitute these estimations and improve the diagnostic accuracy with more dynamic estimations. Our goals are: •Predict defects from Software repositories by learning what are the most important features to analyze and create a prediction model based from existing techniques; •Improve diagnostic accuracy of Barinel with defect prediction probabilities. 1.3 Concepts and definitions Software quality is the degree to which the system meets requirements and expectations of users. The main characteristics of product quality, by ISO/IEC 25010, are portability, maintainability, security, reliability, functional stability, performance efficiency, compatibility and usability [ISO11]. 1.3.1 Software testing glossary According to ISTQB 4, the standard glossary used in software testing is: Error A human action that produces an incorrect result; Fault, defect, bug Flaw in a component that causes the system to fail performing its required functions; Failure Deviation of the component from its expected result. 1.3.2 Types of tests Testing is the process, consisting of static and dynamic activities, that involves the planing, preparation and evaluation of software to check if it meets the requirements and to detect defects. Static testing Is the analysis of the system static representation such as source code and documents, improving the internal quality. 4http://istqb.org 2 Introduction Dynamic testing It involves the execution of the system, observing its behavior under test cases, improving external quality. 1.3.3 Defect prediction Defect prediction consists in reliably predicting software defects using information from code metrics, process metrics or previous defects [DLR12]. Some approaches use binary classification, asserting if a component is defective or not, while others rank them by giving a score. 1.3.4 Fault localization Fault localization consists in finding the component that caused the Software to fail. Is one of the most time consuming activities in debugging. Considering this, there is a high demand for making this process automatic, leading to the development of techniques that makes this activity more effective [WD09]. 1.4 Problem statement Considering our goals, we want to answer the following research questions: RQ1 What features should we extract and analyze from Software Repositories to predict defects? RQ2 Can we improve Barinel fault localization technique from Crowbar with the results from defect prediction? 1.5 Contributions This thesis makes the following contributions: •A technique to parse and represent diffs from patches achieving method granularity in Java; •A model to compute defect probabilities; •A framework for mining Software repositories and reporting analytics in a graphical visualization; •A technique to learn the importance of tracked metrics; •A method to evaluate the gain of using defect probabilities in fault localization, namely the Barinel algorithm in Crowbar. 3 Introduction 1.6 Document overview 1. Introduction An introduction to the context, goals and concepts of this thesis. 2. State of the art Current state of the art techniques on Mining Software Repositories and Software Debugging are reviewed, along with example of existing tools. 3. A Technique to Estimate Defect Probabilities It is presented the tool created to conduct this research and the methodologies used. 4. Experimental results The experimental setup and results are presented in this chapter. 5. Discussion A chapter dedicated to the discussion of findings about the results. Initial research questions are answered. 6. Conclusions and Further Work The conclusions and satisfaction of goals are discussed, along with an overview of further work. 4 Chapter 2 State of the art In this chapter it is described the current state of the art in Mining Software Repositories (MSR), by reviewing findings and discussing existing defect prediction techniques. Related tools are presented and Crowbar, the fault localization tool that we aimed to improve with the defect prediction results. 2.1 Best practices and recommendations Hemmati et al. analyzed the last decade of mining software repositories publications, between 2004 and 2012, and produced an article with best practices and recommendations to new researchers in the MSR community [HNB+13]. The main activities of Mining Software Repositories research are: data extraction and preparation, synthesis, analysis and sharing results. 2.1.1 Data extraction and preparation In data extraction the source code is the most important artifact, although communication artifacts can be used such as emails and issue trackers like Bugzilla. The problem in this phase is the use of wrong assumptions since it is important to understand how the Control Version System is used, the project and its domain since exists noisy data [HJZ12]. For example, a commit message may not reflect the change and can be used only as a way of communication. Also, the study of developers and their behavior can produce valuable information, but the problem of multiple online personas representing the same person should be considered. 2.1.2 Synthesis Synthesis is the phase that involves the prediction and machine learning algorithms that are feed from the extracted data. If we are doing regression analysis, that is estimating the relationship of variables, it is important to be aware of the assumptions used. 5 State of the art Fixcache updates the cache when the bug is fixed and Bugcache when the bug is introduced. The bug-fixing changes are detected by mining the repository commits and bugs database. Bugintroducing changes are detected by the bug-fixing changes. For example, if file A has been fixed in a certain commit, the commit that created this file is a bug-introducing change. Due the presence of false positives, this technique can be improved by using bug databases. A study concluded that Fixcache not only can do bug prediction in a certain point of time but can predict with accuracy at weekly intervals [SLL+11]. 2.3.5 Change Classification Change Classification model evaluates if a change will introduce a bug using machine learning techniques by learning from previous bugs [KWZ08]. A problem that may arise is the effect of noise in the training set, compromising the recall and recognition of buggy changes, but techniques are available to reduce this noise. The classifier decides if a change is buggy or clean with 78% accuracy and 60% percent bug recall on average [KWZ08]. This technique has less prediction power but knowing that a commit introduced a bug it is still useful. The main characteristics are: •classify changes at the file level as buggy or clean; •detect when a bug is introduced and not when it is fixed; •takes advantage of source code information (features); •independent of the programming language using bag-of-words methods. Support Vector Machines is the approach used to classify changes due its performance in text classification applications. Change Classification can be used as a commit checker, a bug indicator on source code editing and can change the software engineering process by giving immediate feedback to trigger a code inspection when a commit is made [KWZ08]. Some limitations are important to consider such as the process of extracting features that depends how developers used Git and like other machine learning approaches, it takes time to learn. 2.4 Fault localization techniques Fault localization is the process of finding the component causing the software execution deviating from its expected result. Traditionally, developers use manual techniques just as injecting prints to debug values or breakpoints. Automatic fault localization techniques are used to reduce the cost of finding these faulty components. 2.4.1 Program-spectra based Program-spectra based methods evaluates the probability of each component being faulty by analyzing the program execution history [PAW14]. It is a statistical technique that, for each test case, 12 State of the art computes the spectrum, that is the code coverage and execution result. The program spectrum is then the list of test cases execution results and can be easily understood by the example in table 2.3. Test Case Component A Component B Component C Result T1 X Success T2 X X Failure T3 X Failure T4 X X Success Table 2.3: Program spectrum To determine what components are called in each test case it is used code instrumentation. The program spectrum in table 2.3 use binary flags so it is called hit spectra. From this input, the components that most affects program execution are computed, calculating similarity coefficients using the Ochiai formula [AZG09]. This technique also exploits information from execution outcomes. The output is then the similarity coefficient for each component, that is their likelihood of containing the fault. Spectrum Fault Localization (SFL) approaches have a good quality of diagnosis and scale well, but is more accurate when the system have many test cases [MS08]. There are some tools based on SFL such as: Gzoltar An eclipse plugin7that integrates with JUnit tests [CRPA12]. Tarantula A technique and a tool8that offers recommendations to reduce the time needed to find the fault [JH05]. 2.4.2 Model-based Model-based techniques use reasoning to do fault localization by having the system knowledge a priori. The system model, the description of correct behavior, is used to compare with the observed behavior of the program and then the difference is used to identify components that explain this deviation [MS08]. Since model-based approaches in software engineering require a formal specification, to avoid this limitation the model is obtained by inference from test cases [PAW14]. The type of model can be: •based on dependencies between program statements; •based on computing values propagation; 7http://gzoltar.com/ 8http://spideruci.org/fault-localization/ 13 State of the art •based on creating abstraction models for particular fault assumptions. Model-based fault localization should be combined with others techniques since the current approaches are not efficient, with high computational cost and scale poorly. 2.4.3 Barinel Barinel is a combination of Spectrum-based fault localization and Model Based Diagnostic [AZG09]. It starts by receiving a hit-spectra matrix, that contains the observation of running the test cases. obs c1c2c3e t11 1 0 1 t20 1 1 1 t31 0 0 1 t41 0 1 0 Figure 2.2: Hit-spectra matrix example Figure 2.2 shows an example of a hit-spectra matrix, with the outcome eof every test case t and the components involved. For example, test case t1hits components {c1,c2}and fails. The algorithm then takes the following steps: Candidate generation Only minimal candidates are generated. A candidate dis a set of components that explains the observed behaviour of the program. In this example, the list of candidates are: •d1={c1,c2} •d2={c1,c3} Candidate ranking Each candidate dis evaluated by computing the posterior probability using the Naïve Bayes rule: Pr(d|obs,e) = Pr(d)·∏ i Pr(obsi,ei|d) Pr(obsi)(2.5) The denominator Pr(obsi)is a term that is normalized for all candidates and it is not used for ranking. Let pjdenote the prior probability of a component being faulty. Then, the prior Pr(d)of a candidate dis: Pr(d) = ∏ j∈d pj·∏ j/∈d (1−pj)(2.6) 14 State of the art Let gjdenote the probability of a component behaving normally (goodness). Then Pr(obsi,ei| d)is computed by: Pr(obsi,ei|d) =        ∏ j∈(d∩obsi) gjif ei=0 1−∏ j∈(d∩obsi) gjotherwise (2.7) If for a certain component gjis not available, it is computed by maximizing Pr(obs,e|d) (Maximum Likelihood Estimation (MLE)), for the Naïve Bayes classifier. Considering our example, the probabilities for both candidates d1and d2are: Pr(d1|obs,e) = Pr(d) z }| { 1 1000 ·1 1000 ·1−1 1000× Pr(obs,e|d) z }| { (1−g1·g2) | {z } t1 ×(1−g2) | {z } t2 ×(1−g1) | {z } t3 ×g1 |{z} t4 (2.8) Pr(d2|obs,e) = Pr(d) z }| { 1 1000 ·1 1000 ·1−1 1000× Pr(obs,e|d) z }| { (1−g1) | {z } t1 ×(1−g3) | {z } t2 ×(1−g1) | {z } t3 ×g1·g3 |{z} t4 (2.9) By performing MLE for both functions: •Pr(d1|obs,e)is maximized for g1=0.47 and g2=0.19; •Pr(d2|obs,e)is maximized for g1=0.41 and g3=0.50. Applying the computed values for goodness, Pr(d1|obs,e) = 1.9×10−9and Pr(d2|obs,e) = 4.0×10−10. The ranking is then (d1,d2). 2.4.4 Crowbar Crowbar9, formerly known as Gzoltar, is a tool for Java projects that relies on test cases (dynamic analysis) to help developers locate where is the fault of a bug. It uses the Barinel algorithm, combining Spectrum-Based Fault Localization and Model-Based approaches. It supports granularity until the statement level and use code instrumentation by injecting probes in the source code. 9https://crowbar.io 15 State of the art Figure 2.3: Crowbar sunburst chart report Figure 2.3 shows an example of a sunburst report that can be zoomed. In this type of visualization, the granularity of components increases from the center to the exterior (statement) and the colors are a clue to find faulty components. We also have the possibility to visualize the report as a vertical partition. Usage Crowbar supports Junit10 and TestNG11. It runs as a Java agent on the test suite through the Maven12 Surefire Plugin13. Configuration is done in the file pom.xml14, that contains information for Maven about the project and configuration details to build and test. Crowbar will then display the results on a web server by displaying an URL that we must access on a browser to be able to see the report. 2.5 Similar tools There are plenty of tools available that analyse Software projects. They evaluate code quality and warn developers about problems detected. 10http://junit.org/ 11http://testng.org/ 12https://maven.apache.org/ 13https://maven.apache.org/surefire/maven-surefire-plugin/ 14https://maven.apache.org/guides/introduction/introduction-to-the-pom.html 16 State of the art 2.5.1 Codacy Codacy 15 is an automatic software revision service (static analysis) that uses defect prediction models to estimate software component reliability. It is a Startup based in Lisbon that won the best pitch award in 2014 Web Summit in London and it is available with free (Open Source) and paid plans. This service uses the Change Classification principle, classifying a commit as buggy or clean. It supports Scala, Javascript, Python, PHP and CSS. With a few steps it is possible connecting our repository, hosted at Bitbucket or Github. Figure 2.4: Codacy dashboard Figure 2.4 shows the dashboard with a variety of metrics, reporting a score for code style, errors, code complexity, performance, unused code, compatibility, etc. New and fixed issues are presented, therefore, developers feel rewarded and motivated to improve code quality, a feature that somehow failed in the tool developed in a research conducted at Google in 2013 by Chris Lewis et al. [LLS+13]. 2.5.2 Moskito Moskito16 is a tool that monitors Java Web applications, does fault localization and it is free and Open Source 17. Developers must use annotations to declare what classes or methods to monitor and it does not require changing code, which make this solution simple to use. Two main elements 15https://codacy.com 16http://moskito.org 17https://github.com/anotheria/moskito-control 17 State of the art of this tool are the agent and server, where the first collects data and sends it to the server, that processes and displays information in a dashboard. 1//simply add @Monitor 2@Monitor 3public class MonitoredClass { 4public void firstMethod(){ 5//do something 6} 7public void secondMethod(){ 8//do something else 9} 10 //you can also exclude methods from monitoring: 11 @DontMonitor 12 public void doNotMonitorMe(){ 13 } 14 } Listing 2.1: Usage example from Moskito documentation Figure 2.5: Moskito dashboard The dashboard at figure 2.5 displays performance charts taken from multiple nodes. When a component performance changes, the health indicator change its color, so developers can fix the problem immediately, before affecting the whole application and users complain. 2.5.3 SensioLabsInsight SensioLabsInsight in figure 2.6, is a web service 18 that continuously analyzes PHP projects in terms of security, bugs and other quality checks. It also does dynamic analysis to improve di18https://insight.sensiolabs.com/ 18 State of the art agnostic accuracy. Integrates with Github and Bitbucket services and is free for Open Source projects. Figure 2.6: Example of a report from SensioLabsInsight 2.5.4 Code Climate Code Climate 19 in figure 2.7, is another service that analyzes PHP, Python, Javascript and Ruby using static analysis. It essentially produce warnings about issues in code complexity, duplication, style and readability. Also displays insights about churn (lines changed) versus code quality. The dashboard is very complete since we can check listed issues or inspect code along with the warnings produced. It has a free plan for Open Source projects. 19https://codeclimate.com/ 19 State of the art Figure 2.7: Issues listed on Code Climate 2.5.5 Pull Review Pull Review 20 in figure 2.8, is an automatic code review service (static analysis) just for Ruby. It gives feedback for style, duplication, code smells, documentation, security and tests. It has the ability of linking a Github, Gitlab or Bitbucket repository and is also free for Open Source. Figure 2.8: Pull Review report 20https://pullreview.com/ 20 Chapter 3 A Technique to Estimate Defect Probabilities In this chapter it is discussed the methodologies used to answer the research questions. 3.1 Schwa Schwa is the tool developed for research. Source code is available at Github1as an Open Source project under MIT 2.0 license. It was developed in Python and is hosted at PyPi2, the Python Package Index. 3.1.1 Installation Schwa relies on Python 3 and Git and they are the only dependencies. Can be easily installed using pip 3. 1pip3 install schwa --pre Listing 3.1: Schwa installation command 3.1.2 Usage It can be used as a command line tool and imported as a Python package. An example of running Schwa, is analyzing the last 20 commits of the Joda Time repository: 1https://github.com/andrefreitas/schwa 2https://pypi.python.org/pypi/Schwa 3https://pip.pypa.io/en/latest/installing.html 21 A Technique to Estimate Defect Probabilities constraints(R,F,A) = (R+F+A) == 1∧(R∗F∗A)>0 (3.6) distance(R,F,A,m) = ∑c∈involved(m)score(R,F,A,c) |involved(m)|−∑c∈(Components\involved(m)) score(R,F,A,c) |(Components \involved(m))| (3.7) score(R,F,A,c) = crevisions ∗R+cfixes ∗F+cauthors ∗A(3.8) This means that in bug-introducing commits, faulty components should have higher score than non faulty components. Besides finding features weights, this approach helps us validate if Schwa is predicting defects correctly. Due to performance constraints, the fitness function only evaluates components at the file granularity. Note that this granularity is only for Schwa learning mode. For defect prediction, Schwa still operates with method granularity for Java. 3.3 Diagnostic cost Crowbar outputs a rank of components ordered by their probability of having the fault. To measure the quality of this diagnostic, the heuristic used is Diagnostic Cost [CAFd13] that is the average of the minimum and maximum distance that faulty components are from the top of the rank. For multiple faults, this cost is computed considering the lowest faulty component. diagnosticcost =mindistance +maxdistance −|f aulty| 2(3.9) 3.3.1 Example C2is the component that have the fault. Position Rank 1 Rank 2 1C3(0.234)∗C2(0.500) 2C5(0.145)C5(0.120) 3∗C2(0.145)C3(0.120) Diagnostic Cost 1 0 Table 3.1: Examples of ranks In Rank 1 C5and C2components have the same probability so C2position can be 2 or 3 depending on the result of ordering the rank. In Rank 2 C2is on the first position so the cost is 0. 28 A Technique to Estimate Defect Probabilities 3.3.2 Schwa integration with Crowbar By using Schwa, the goal is improving Barinel results, reducing the Diagnostic Cost by evaluating if defect probabilities should be used in priors (pj), goodness (gj) or both parameters. In Barinel the meaning of these parameters is the following: Prior The probability a component have of causing an error and is by default 1 1000 , based on the heuristic that for every 1000 lines of code heuristic there is one bug. Goodness The probability of a component behaving normally and by default is computed by using the Maximum Likelihood Estimation that is heavy to compute. By default is 0.5. If the defect probability of Schwa is used for goodness, it is computed by: Pr(obsi,ei|d) = 1−de f ectprobability(c)(3.10) 29 A Technique to Estimate Defect Probabilities 30 Chapter 4 Experimental results This chapter provides the results of the experiments conducted. The setup and configurations are also described. There is a public page with the experimental results of Schwa on Github 1. 4.1 Features weight estimation In this section we present the results of learning the features weights, that is the importance of each of them, when computing the defect probability. 4.1.1 Experimental setup To run this experiment, Schwa was invoked in the learning mode with 5, 50 and 100 commits with the script in listing 4.1. The version of Schwa used was 0.1.dev24(tagged on Git). Running genetic algorithms takes a substantial time of computation, so we have used Crowdsourcing to run the experiment in a variety of projects. We collected data from academic, enterprise and Open Source projects to have results from different contexts. 1if [ -z "$1" ] 2then 3echo "usage: $0 <git repository path>" 4else 5touch report_$USER.txt 6echo "This will take a while..." 7echo "Learning with 5 commits" 8schwa $1 --commits 5 -l >> report_$USER.txt 9echo "Learning with 50 commits" 10 schwa $1 --commits 50 -l >> report_$USER.txt 11 echo "Learning with 100 commits" 1https://github.com/andrefreitas/schwa/wiki/Experiments 31 Experimental results 12 schwa $1 --commits 100 -l >> report_$USER.txt 13 echo "Thank you! You are the best! Send report_$USER.txt to Andre :)" 14 fi Listing 4.1: Shell script used to learn features weights 4.1.2 Results Here we show the results for each repository. In the following tables, in each row we have the maximum number of commits, the weights of revisions, fixes and authors and the fitness value. The main objective of these tables is providing the importance of each feature for a certain project. 4.1.2.1 Schwa The experiment was run in Schwa2itself, that have only one contributor. It is developed mostly with Python along with HTML, CSS and Javascript. Commits Revisions Fixes Authors Fitness 5 0.2857 0.2857 0.4286 0 50 0.7143 0.1429 0.1429 1.0699 100 0.7143 0.1429 0.1429 1.1875 Table 4.1: Schwa results For 5 commits, the fitness function is 0, so it is not possible to estimate correctly the weights. For 50 and 100 commits, revisions is the most important feature. 4.1.2.2 Libcrowbar Libcrowbar is the main repository of Crowbar and have multiple academic contributors. The technologies used are Java, C++, HTML, CSS and Javascript. Commits Revisions Fixes Authors Fitness 5 0.1429 0.1429 0.7143 0.3486 50 0.7143 0.1429 0.1429 1.3639 100 0.7143 0.1429 0.1429 0.3387 Table 4.2: Libcrowbar results For the last 5 commits, authors is the most important feature and for 50 and 100 commits, is revisions. 2https://github.com/andrefreitas/schwa 32 Experimental results 4.1.2.3 Joda Time Joda Time3is an Open Source time library for Java with multiple contributors. Commits Revisions Fixes Authors Fitness 5 0.1429 0.2857 0.5714 0 50 0.1429 0.7143 0.1429 -2.1874 100 0.1429 0.7143 0.1429 0.3893 Table 4.3: Joda Time results For this project only for 100 commits the fitness function is > 0, showing that fixes is the most important feature. 4.1.2.4 Meo Arena Meo Arena4is a mobile application developed for Android in an academic context with two contributors. It relies on Java and Python. Commits Revisions Fixes Authors Fitness 5 0.2857 0.4286 0.2857 0 50 0.7143 0.1429 0.1429 1.6712 100 0.7143 0.1429 0.1429 0.7368 Table 4.4: Meo Arena results For 5 commits, fitness is equal to zero. For 50 and 100 commits the most important feature is revisions. 4.1.2.5 Mongo Java Driver Mongo Java driver5is an Open Source projects and enables Java applications to communicate with a MongoDB server. It is developed with Java and Groovy and has multiple contributors. Commits Revisions Fixes Authors Fitness 5 0.7143 0.1429 0.1429 0.3486 50 0.7143 0.1429 0.1429 0.1685 100 0.1429 0.7143 0.1429 1.4666 Table 4.5: Mongo Java driver results For 5 and 50 commits the most important feature is revisions but for 100, is fixes. 3https://github.com/JodaOrg/joda-time 4https://bitbucket.org/andrefreitas/feup-cmov-meoarena/ 5https://github.com/mongodb/mongo-java-driver 33 Experimental results 4.1.2.6 Scraim Scraim6is a web-based project management tool developed by the company Strongstep. It is build on top of Redmine and uses Ruby on Rails. Commits Revisions Fixes Authors Fitness 5 0.4286 0.1429 0.4286 0.3486 50 0.1429 0.1429 0.7143 0.3146 100 0.1429 0.7143 0.1429 0.9399 Table 4.6: Scraim results For 5 commits, revisions and authors are both important. For 50, authors is the most important and for 100 is fixes. 4.1.2.7 Trainsim Trainsim is an academic project developed with Java and Swing with one contributor. Commits Revisions Fixes Authors Fitness 5 0.1429 0.7143 0.1429 0 50 0.7143 0.1429 0.1429 1.5945 100 0.5714 0.2857 0.1429 2.0425 Table 4.7: Trainsim results For 5 commits fitness is zero, for 50 and 100 the most important is revisions. Fixes importance increased from 50 to 100 commits. 4.1.2.8 ShiftForward ShifForward7is a company that develops advertising technology with Scala. They have contributed with results of three projects. Commits Revisions Fixes Authors Fitness 5 0.4286 0.1429 0.4286 0 50 0.4286 0.4286 0.1429 1.9044 100 0.2857 0.4286 0.2857 3.3134 Table 4.8: Adstax results In the Adstax project, for 5 commits the fitness is zero. For 50 commits, revisions and fixes are both important and for 100, fixes is the most important. 6https://scraim.com/ 7http://shiftforward.eu 34 Experimental results Commits Revisions Fixes Authors Fitness 5 0.2857 0.1429 0.5714 0.4408 50 0.1429 0.1429 0.7143 0.4457 100 0.1429 0.7143 0.1429 -1.4159 Table 4.9: Boxer results In the Boxer project, for 5 and 50 commits, authors is the most important feature. For 100 commits, the fitness is negative. Commits Revisions Fixes Authors Fitness 5 0.7143 0.1429 0.1429 0.9439 50 0.1429 0.7143 0.1429 1.9547 100 0.1429 0.4286 0.4286 1.3927 Table 4.10: Apso results In the project Apso, revisions is the most important feature for 5 commits. For 50 commits is fixes and for 100 commits is fixes and authors. 4.1.2.9 Hivedb Hivedb8is an Open Source framework for horizontally partitioning MySQL systems. It is developed mostly in Java with multiple contributors. Commits Revisions Fixes Authors Fitness 5 0.7143 0.1429 0.1429 0 50 0.1429 0.7143 0.1429 0.1739 100 0.7143 0.1429 0.1429 0.9095 Table 4.11: Hivedb results For 5 commits the fitness function is zero. For 50 commits the most important feature is fixes and for 100, is revisions. 4.1.2.10 Teamengine Teamengine9is an Open Source engine to test web services and other resources in Java. Commits Revisions Fixes Authors Fitness 5 0.7143 0.1429 0.1429 0 50 0.1429 0.7143 0.1429 1.1356 100 0.7143 0.1429 0.1429 3.1080 Table 4.12: Teamengine results 8http://hivedb.org 9https://github.com/opengeospatial/teamengine 35 Experimental results For 5 commits the fitness function is 0. Fixes is the most important feature for 50 commits and for 100 is revisions. 4.1.2.11 Automatalib Automatalib10 is an Open Source Java Library for representing automata, graphs and transition systems. Commits Revisions Fixes Authors Fitness 5 0.1429 0.1429 0.7143 0.3486 50 0.1429 0.7143 0.1429 -0.1952 100 0.4286 0.4286 0.1429 0.5261 Table 4.13: Automatalib results For 5 commits, authors is the most important feature. For 50 commits the fitness is negative. Revisions and fixes have the same weight for 100 commits. 4.1.2.12 CDI TCK CDI TCK11 is an Open Source Context and Dependency Injection for Java EE developed with Java. Commits Revisions Fixes Authors Fitness 5 0.1429 0.5714 0.2857 0 50 0.2857 0.2857 0.4286 0 100 0.1429 0.7143 0.1429 -0.3247 Table 4.14: CDI TCK results For 5 and 50 commits fitness is zero and for 100 is negative. 4.2 Diagnostic cost In this section, the results of computing the diagnostic cost for each configuration of Schwa in Crowbar are presented. In this experiment, the computational cost is also substantial and crowdsourcing could not be used since we need to inspect the behaviour of Crowbar with Schwa. Considering that these experiments were run in a laptop (can last hours) with the constraints of finding Java projects that use Git, have tests and support Maven, this phase took weeks to find conclusive results. Joda Time and CDI TCK projects were selected to run these experiment since they have a Git repository and are Java projects with tests, so they both can be used with Crowbar. 10https://github.com/misberner/automatalib 11https://github.com/cdi-spec/cdi-tck 36 Experimental results 4.2.1 Experimental setup For each project we had setup the experimental environment with the following steps: •Compute the weights for revisions, fixes and authors with Schwa learning mode; •Create a .schwa.yml in the root of the repository with the weights and maximum commits; •Insert bugs (e.g. wrong comparison) in methods and commit the changes; •Evaluate the diagnostic cost for using Schwa with priors, goodnesses or both. 4.2.2 Results The results are presented with the history of commits and configurations of Schwa. 4.2.2.1 Joda Time The sequence of commits applied in Joda Time is available on table 4.15 along with the commits that inserted bugs. Order Commit Description 1 8207a55 Added a defect in DateTime.java in withZoneRetainfields() 2 74149c0 Added a defect in Duration.java in minus() 3 22a5f71 Fixed withZoneRetainfields() bug 4 0945c34 Fixed minus() bug and added another bug 5 92adf94 Fixed previous bug and added one in getMaximumValue() Table 4.15: Commits applied to Joda Time 1public DateTime withZoneRetainFields(DateTimeZone newZone) { 2newZone = DateTimeUtils.getZone(newZone); 3DateTimeZone originalZone = DateTimeUtils.getZone(getZone()); 4-if (newZone == originalZone) { 5+if (newZone != originalZone) { 6return this; 7} Listing 4.2: Commit 8207a55 patch 1public Duration minus(long amount) { 2-return withDurationAdded(amount, -1); 3+return withDurationAdded(amount, -2); 4} 5 6public Duration minus(ReadableDuration amount) { 37 Discussion 5.2 Diagnostic cost The results from diagnostic cost experiments indicate that we could not improve the results of Crowbar but found an alternative way of estimating defect probabilities in the Barinel technique: Improvement of diagnostic results We could not find an example of Schwa improving the diagnostic results of Crowbar. But, we must note that even with optimal defect predictions results from Schwa, in some cases the diagnostic cost cannot be improved, as seen in Joda Time. Importance of recently changed components In the first results from Joda Time we were getting worse results because faulty components that had been recently changed, had low defect probability. By modifying the TWR function with the Time Range parameter, when Schwa was used in priors, it did not got worse results. Faster defect prediction results with Schwa Since the Barinel algorithm uses the MLE to estimate goodnesses and priors, this process can take for example 2 hours in some cases. By using Schwa, we reduced this phase to less than 1 minute. Computational power A cluster is better suited than a laptop to get results in a more convenient time. Schwa is I/O intensive because it is parsing and extracting code from commits. Crowbar have a substantial time complexity by running the MLE algorithm and can benefit of faster CPUs. 5.3 Threats to validity Regarding the experiments for estimating features weight, the usage of 3 bits for representing the weights of individuals can limit the possibility of searching better solutions. For the diagnostic cost, the results are just from two projects that are open source. 44 Chapter 6 Conclusions and Further Work We have developed a framework capable of predicting software defects from repositories, with a web-based graphical report. The creation of a learning mode for Schwa with genetic algorithms, gives researchers the ability of evaluating new features to extract from repositories, making Schwa a convenient framework to study Mining Software Repositories. Schwa should be combined with other techniques, since it is not completely accurate. Code review is an example of an activity that can benefit from this tool, allowing developers to focus in the most important components. The usage of Python allowed a fast prototyping of ideas due its simplicity and the existing of useful libraries. Mining Software Repositories is a time-consuming activity so research in this subject can benefit from the usage of clusters. 6.1 Goals satisfaction We successfully created a defect prediction technique based on MSR approaches capable of learning features, until the method granularity for Java projects. Our initial goal of generalizing features weights was refuted by the experimental results, that shown that for each projects they are different. Although we did not improve the accuracy of Barinel, we have come with an alternative technique of computing defect probabilities in less time. For example, since Barinel for Joda Time can take 2 hours to run MLE, now with Schwa, this phase takes less that 1 minute, so it is a substantial achievement. 6.2 Further work The technique used in Schwa for learning features can be improved with optimizations in the binary representation and code parallelization. There are plenty of improvements that can be done 45 Conclusions and Further Work in Schwa: •Support of more programming languages; •Improve performance on extraction by developing a Python module in C; •Add charts for revisions, fixes and authors evolution in the visualization, to support the results with more reasoning; •Develop a SaaS platform for Schwa, similar to Codeclimate and Codacy. MSR research could benefit of new techniques that reduce noise in the classification of bugfixing commits, that can exploit issue trackers. Schwa could benefit from reducing this noise. With more computational power, we could evaluate with more examples, the gain of using Schwa in Crowbar, by finding an example where the diagnostic cost decreased. 46 References [AZG09] Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund. Spectrum-based multiple fault localization. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, ASE ’09, pages 88–99, Washington, DC, USA, 2009. IEEE Computer Society. [CAFd13] J. Campos, R. Abreu, G. Fraser, and M. d’Amorim. Entropy-based test generation for improved fault localization. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 257–267, Nov 2013. [Car13] Emil Carlsson. Mining git repositories : An introduction to repository mining, 2013. Linnaeus University, Department of Computer Science. Degree of Bachelor. [CK94] S.R. Chidamber and C.F. Kemerer. A metrics suite for object oriented design. Software Engineering, IEEE Transactions on, 20(6):476–493, Jun 1994. [CRPA12] José Campos, André Riboira, Alexandre Perez, and Rui Abreu. GZoltar: an Eclipse plug-in for Testing and Debugging. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pages 378–381, New York, NY, USA, 2012. ACM. [DLR12] Marco D’Ambros, Michele Lanza, and Romain Robbes. Evaluating defect prediction approaches: A benchmark and an extensive comparison. Empirical Softw. Engg., 17(4-5):531–577, August 2012. [FN99] N.E. Fenton and M. Neil. A critique of software defect prediction models. Software Engineering, IEEE Transactions on, 25(5):675–689, Sep 1999. [GKMS00] T.L. Graves, A.F. Karr, J.S. Marron, and H. Siy. Predicting fault incidence using software change history. Software Engineering, IEEE Transactions on, 26(7):653– 661, Jul 2000. [HJZ12] Kim Herzig, Sascha Just, and Andreas Zeller. It’s not a bug, it’s a feature: How misclassification impacts bug prediction. Technical report, Universität des Saarlandes, Saarbrücken, Germany, August 2012. [HNB+13] Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko, Wei Wang, Reid Holmes, and Michael W. Godfrey. The MSR cookbook: Mining a decade of research. In Proceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, pages 343–352, Piscataway, NJ, USA, 2013. IEEE Press. [ISO11] ISO. Systems and software engineering – systems and software quality requirements and evaluation (square) – system and software quality models. ISO ISO/IEC 47 REFERENCES 25010:2011, International Organization for Standardization, Geneva, Switzerland, 2011. [JH05] James A. Jones and Mary Jean Harrold. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 273–282, November 2005. [KWZ08] Sunghun Kim, E. James Whitehead, Jr., and Yi Zhang. Classifying software changes: Clean or buggy? IEEE Trans. Softw. Eng., 34(2):181–196, March 2008. [KZWJZ07] Sunghun Kim, Thomas Zimmermann, E. James Whitehead Jr., and Andreas Zeller. Predicting faults from cached history. In Proceedings of the 29th International Conference on Software Engineering, ICSE ’07, pages 489–498, Washington, DC, USA, 2007. IEEE Computer Society. [LLS+13] Chris Lewis, Zhongpeng Lin, Caitlin Sadowski, Xiaoyan Zhu, Rong Ou, and E. James Whitehead Jr. Does bug prediction support human developers? findings from a google case study. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pages 372–381, Piscataway, NJ, USA, 2013. IEEE Press. [MPS08] Raimund Moser, Witold Pedrycz, and Giancarlo Succi. A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, pages 181–190, New York, NY, USA, 2008. ACM. [MS08] W. Mayer and M. Stumptner. Evaluating models for model-based debugging. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, ASE ’08, pages 128–137, Washington, DC, USA, 2008. IEEE Computer Society. [PAW14] Alexandre Perez, Rui Abreu, and Eric Wong. A survey on fault localization techniques. 2014. Technical report. [Ser13] F. Servant. Supporting bug investigation using history analysis. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 754–757, Nov 2013. [SJ12] Francisco Servant and James A. Jones. History slicing: Assisting code-evolution tasks. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12, pages 43:1–43:11, New York, NY, USA, 2012. ACM. [SLL+11] Caitlin Sadowski, Chris Lewis, Zhongpeng Lin, Xiaoyan Zhu, and E. James Whitehead, Jr. An empirical analysis of the fixcache algorithm. In Proceedings of the 8th Working Conference on Mining Software Repositories, MSR ’11, pages 219–222, New York, NY, USA, 2011. ACM. [WD09] W. Eric Wong and Vidroha Debroy. A survey of software fault localization, 2009. Technical report. 48 REFERENCES [WH05] Chadd C. Williams and Jeffrey K. Hollingsworth. Automatic mining of source code repositories to improve bug finding techniques. IEEE Trans. Softw. Eng., 31(6):466– 480, June 2005. [ZPZ07] Thomas Zimmermann, Rahul Premraj, and Andreas Zeller. Predicting defects for eclipse. In Proceedings of the Third International Workshop on Predictor Models in Software Engineering, PROMISE ’07, pages 9–, Washington, DC, USA, 2007. IEEE Computer Society. 49