A Family of Experiments to Evaluate the Effects of Mindfulness on Software Engineering Students: The MetaMind Dataset

1 Information Classification: General A Family of Experiments to Evaluate the Effects of Mindfulness on Software Engineering Students: The MetaMind Dataset Beatriz Bernárdez, Margarita Cruz, Amador Durán, José A. Parejo, Antonio Ruiz-Cortés Universidad de Sevilla, Spain {beat, cruz, amador, japarejo, aruiz}@us.es Abstract. Context: Software Engineering students are often excellent developers although they may occasionally encounter difficulties with certain tasks such as conceptual modelling, in which mental clarity plays an essential role. With several years of experience in practising mindfulness (a meditation technique that calm the mind to see with clarity), we hypothesise that several weeks of continued mindfulness practise may increase the performance of students regarding conceptual modelling since the proven benefits of mindfulness include increased concentration and mental clarity. In order to ascertain this hypothesis, a family of controlled experiments, involving 130 students, has been carried out at the University of Seville over three consecutive academic years. Subsequent to the analysis of the individual experiments, a meta—analysis was conducted. Aim: This chapter helps understand how various datasets have been generated and traces their evolution across the family of experiments and the resulting meta—analysis. Not only does the process include the integration of datasets into a single target dataset with the complete set of data, but it also improves the structure of the dataset and adapts it to several statistical analysis tools. Method: Data collection was manually carried out during the individual experiments by means of gathering the scores of the students attained in conceptual modelling exercises. In order to obtain the complete but also simple dataset of the family of experiments, certain dataset columns were necessarily selected and renamed. New relevant information was also added to form a conclusive meta—analysis. The dataset was adapted to the type and format of SPSS for the execution of a further analysis, complementary to the main study in R. The MetaMind dataset is the set of datasets involved in the present research. The MetaMind dataset has become a helpful resource for the independent reproducibility of the results, to ascertain the evolution of the process across a family of experiments, and to enable external replications of the controlled experiment. Keywords family of experiments, controlled experiments, software engineering students, mindfulness, meta—analysis. Background & Summary Mindfulness is a meditation technique whose main goal involves maintaining a calm mind and training attention by focusing only on one single thing (the support) at a time, this support is usually the practitioner’s breathing. During a session of mindfulness, practitioners remain in a quiet location, programme an alarm, and focus their attention on the support. When other thoughts come to mind, they are simply acknowledged, set aside, and the practitioners return their attention to their breathing. 2 Information Classification: General Improved attention, concentration, and mental clarity have been shown to be present among the benefits of mindfulness [1]. The reported benefits of its continued practice can be of interest for Software Engineering students and practitioners, especially in tasks such as conceptual modelling, in which concentration and clearness of mind are crucial. In order to ascertain whether the practice of mindfulness improves the students’ performance in conceptual modelling tasks, a family of three controlled experiments have been carried out in the Software Engineering faculty of the University of Seville, Spain. The idea of the original experiment (Mind#1) is to evaluate whether second-year Software Engineering students improved their performance in conceptual modelling subsequent to the daily practice of mindfulness over a period of four weeks, by comparing their results with those of a control group receiving a placebo treatment. In order to achieve this goal, in Mind#1, the students were split into two groups: experimental and control. The experimental group was involved in a mindfulness workshop for four weeks. All the students developed two conceptual modelling exercises 1 , one before starting the mindfulness workshop and a second when the workshop had finished. The students’ evolution was analysed in terms of effectiveness (the rate of correctly identified model elements) and efficiency (the number of correctly identified model elements per unit of time). The results showed that students who practised mindfulness were statistically significantly more efficient than those of the control group. With respect to effectiveness, several improvements were observed but were not statistically significant [2]. Mind#1 was executed during the 2013–2014 academic year. Two internal replications (Mind#2 and Mind#3) were carried out in successive academic years with a double purpose: to confirm the results of Mind#1, and to overcome the limitations of the original experimental design [3]. One of the main limitations was related to the conceptual modelling exercises: after analysing Mind#2, it was detected that certain unnoticed characteristics of such exercises could influence the outcome of the experiments. Therefore, in order to study such impact, it was decided to change the order of the conceptual modelling exercises in Mind#3. Hence, the students carried out the task involving EoDP before the mindfulness workshop and then carried out the task involving ERASMUS after said workshop. Once Mind#3 had finished, in order to analyse the whole family of experiments, a meta—analysis was performed, that is, a pooled data meta–analysis, in which the data of three studies are analysed as if they had originated from a single study, and an aggregated data meta—analysis, which usually focuses on the analysis of the effect size and its variation among the studies. The results of such meta—analysis reported in [4], revealed that both the effectiveness and efficiency of the students who practised mindfulness were statistically significantly better than those of the control group. 1 Two well-known problem domains were chosen for the conceptual modelling task: the first involved ERASMUS grants, and the second involved End of Degree Projects (EoDP). 3 Information Classification: General The present chapter reports the dataset of the family of experiments, called MetaMind, employed to perform such a meta—analysis. The MetaMind dataset was generated as the aggregation of the three datasets of individual experiments (Mind#1, Mind#2, and Mind#3), but included certain new information deemed useful for the formation of a more conclusive meta—analysis. Figure 1 shows the process of collecting, transforming, and analysing the MetaMind dataset. Initially, for Mind#1, Mind#2 and Mind#3, the data collection was manually recorded using spreadsheets. After that, a filtered csv file was subsequently generated from these three spreadsheets. All the csv files were then automatically processed for their integration into a single file containing the simplified dataset of the whole family of experiments. During this process, information regarding the academic year (2014, 2015, or 2016) and the order in which the exercises were carried out in each individual experiment were added to the dataset. The csv file was used for the analysis in R-Studio. Furthermore, in order to enable most of the analyses to also be executed in SPSS, the data was imported to SPSS and saved as a file with extension “.sav”, that is, the data file type in SPSS. Figure 1. Process of Collection and Transformation of the MetaMind Dataset Reported in this Chapter. 4 Information Classification: General Dataset Specification Subject Effect of mindfulness on students’ performance in conceptual modelling task. A Family of experiments. Specific subject area Students’ effectiveness and efficiency in a conceptual modelling task, for both pooled and aggregated meta—analyses of the family of experiments. Type of data csv files and sav files (SPSS). How data were acquired The effectiveness and efficiency scores were acquired manually, by means of a careful reading of the UML class diagrams developed by the students as their solution to the conceptual modelling exercises. After evaluating each student’s solution, each score was labelled either as Pre—Treatment when the solution was obtained before the treatment, or Post—Treatment otherwise. Data format Analysed. Parameters for data collection To participate in the sample, students must fulfil the following selection criteria: (i) to attend at least 75% of the mindfulness sessions; (ii) to perform both ERASMUS and EoDP tasks; and (iii) to employ the correct notation in the conceptual modelling exercises, that is, UML class diagrams. The number of subjects that fulfilled the selection criteria was 36 in Mind#1, 53 in Mind#2, and 45 in Mind#3. As a consequence, there were 130 subjects in the sample for meta—analysis. The experimenters have accorded a reference model for each conceptual modelling exercise (ERASMUS and EoDP). Such models have been employed to evaluate the student’s solution by assigning it a score. Description of data collection Data collection was carried out through the evaluation of the conceptual modelling solutions presented by the students before and after the mindfulness sessions. Model elements in a student’s solution were considered as correctly identified if there existed identical or semantically equivalent counterparts in the corresponding reference model. For the effectiveness response variable, the score is the number of model elements (UML classes, attributes, and associations) correctly identified over the number of elements in the reference model. For the efficiency response variable, the score is the number of model elements correctly identified over the time spent solving the exercise. Files Mind#1: data2014.csv, Mind#2: data2015.csv, Mind#3: data2016.csv Meta—analysis (R-Studio): simplifieddata.csv Meta—analysis (SPSS): fulldata.sav and fulldataforancova.sav Data source location Institution: Applied Software Engineering (ISA) research group. University of Seville City: Seville Country: Spain 5 Information Classification: General Data accessibility Repository name: EXEMPLAR platform described in [5] Direct URL to data: https://exemplar.us.es/demo/BernardezMindfulnessTSE Related research article B. Bernárdez, A. Durán, J. A. Parejo, N. Juristo, A. Ruiz-Cortés, “Effects of Mindfulness on Conceptual Modelling Performance: a Series of Experiments”, IEEE Transactions on Software Engineering. DOI 10.1109/TSE.2020.2991699. In press. Value of the Data • The MetaMind dataset provides the key for the correct understanding not only of the meta— analysis, but also of the process followed leading to its generation from the individual experiment datasets. • The MetaMind dataset offers the possibility of reproduction of the experimental results. The Association for Computing Machinery (ACM) in [6] clearly states the importance held by experimental results that can be independently reproduced. • According to the taxonomy proposed by [6], the MetaMind dataset is labelled as “available” because it has been made permanently available for retrieval. By releasing the dataset in this chapter, we believe that it may be closer to achieving the highest level of replication for the label of “replicated” since, the results of the study have been obtained in a subsequent study by a person or team other than ourselves, having employed, in part, our laboratory package on https://exemplar.us.es/demo/BernardezMindfulnessTSE. • Mind#1 already has two internal replications, however the external replications have to be performed by independent researchers and therefore involve a greater power of confirmation [7]. By releasing the dataset in this chapter, we increase the possibility of external replication, because the laboratory package, including the MetaMind dataset and experimental material, gains visibility. The unavailability of laboratory packages has been identified by the research community as one of the main causes of the low number of replications [8]. 6 Information Classification: General Data Description The MetaMind dataset is available online as part of a complete laboratory package that also includes files of experimental material and statistical analyses. MetaMind is composed of: (i) the files of the three experiments in the family; and (ii) a set of files for the meta—analysis, which are held in various formats, depending on the tool that is applied to process the data, but all with the same content. Table 1 shows the fields of the MetaMind file schema. Subject An anonymous student identifier. Efficiency The score of efficiency for each subject at a particular point in time. Effectiveness The score of effectiveness for each subject at a particular point in time. Year The year of the individual experiment in which the subject had participated: in the case of Mind#1, the year is 2014; for Mind#2, the year is 2015; and for Mind#3, the year is 2016. Group The group to which the subject belongs: either experimental or control. The field has the value MF (MindFulness) if the subject belongs to the experimental group, otherwise the field has the value NULL. Time The moment at which the students developed the exercises. If the measure was taken before the treatment (mindfulness workshop) the field has the value Pre, otherwise the field has the value Post. Order The order in which the conceptual modelling exercises were performed in each experiment. Thus, the order in Mind#1 and Mind#2 has the value ERASMUS→EoDP, but in Mind#3 the order has the value EoDP→ERASMUS. Exercise The specific conceptual modelling exercise in which the student achieves the corresponding score on the response variables. Table 1. Data description of the MetaMind dataset Since most of the statistical tests have been performed in both R and SPSS as double–checking, the dataset records in two different formats: csv and sav. As mentioned earlier, although there are two meta—analysis files, they both hold a common content. On the one hand, simplieddata.csv is the dataset file for the analysis in R-Studio, following the structure in Table 1. This file is automatically generated from the individual experiment dataset files 7 Information Classification: General (data2014.csv, data2015.csv, and data2016.csv), by selecting, renaming the variables, and adding the information regarding order and year. On the other hand, by taking into account that, in SPSS, different statistical tests require a different rawdata format, two dataset formats have been considered. First, fulldata.sav constitutes the file which has the same structure as simplifieddata.csv in that it has 260 rows (two for each subject in the sample). In the second format, the file fulldataforancova.sav follows a wide format in that repeated measure data appears in two columns (Pre–Treatment, Post–Treatment) for each response variable. Experimental Design, Materials, and Methods In order to study the evolution of the students over a period of time, a major requirement for choosing the experimental design involved measuring the conceptual modelling performance of each student before and after the period of practice of mindfulness, by means of a repeated measure design. As a consequence, a within–subjects factor was identified called time, representing the moment in which each conceptual modelling exercises was performed. The factor time has two levels, Pre–Treatment and Post–Treatment. However, the main factor in the family of experiments was a between–subjects factor, called group, which represents the group in which a particular student was assigned. This factor has two levels: the experimental group (students who practise mindfulness), and the control group (to which a placebo treatment was administered). Since this design includes both within and a between–subjects factors, the 2x2 mixed factorial design proposed by [9] was selected. This design is commonly used in the fields of psychology and medicine when the evolution of patients under a given treatment (or therapy) is studied after a certain amount of time. The methodology included the following steps: i) each subject was randomly 1 assigned one single treatment (group); ii) both groups of students developed the first conceptual modelling exercise (Pre– Treatment); iii) the experimental group practised mindfulness over a specified period of time; and iv) both groups of students were subject to the second conceptual modelling exercise (Post–Treatment). Therefore, two repeated measures on the response variables were taken (Pre–Treatment and Post– Treatment). With respect to the experimental material, two slide presentations were designed: i) for the recruitment of students; and ii) for the introduction of students to the practice of mindfulness. Several questionnaires were also used so that the students could express their interest and a demographic study 1 In Mind#1, the subjects chose whether they wanted to practise mindfulness. In accordance with their selection, they were assigned either to the experimental group or to the control group. 8 Information Classification: General of the sample could be carried out. The statements of the conceptual modelling exercises were written as an interview transcript, that is, they contained questions and answers between a requirement engineer and a customer regarding a particular problem domain and the expected system behaviour. As commented above, ERASMUS and EoDP were chosen under the influence of the criteria regarding the familiarity and interest of the students, since these constitute the potential candidates for an ERASMUS grant and they also have to develop an EoDP to finish their degree studies. Furthermore, the experimental material includes the experiment specifications in SEDL (Scientific Experiments Description Language), which is the domain-specific language for the experiment description of the EXEMPLAR platform [5]. In this section the experimental setting of the family of experiment is summarized. A detailed analysis of the threats to the validity of such experimental setting and the actions performed to mitigate them are compiled in [4]. About benefits of mindfulness in Software Engineering activities Several Information Technology organizations have successfully incorporated mindfulness into their teams [10, 11]. Among other advantages, the practice of mindfulness can improve the flow of meetings in agile development teams, in terms of effectiveness, decision-making, and listening, as shown in [12]. Although, in our family of experiments, the most influential benefits of mindfulness are attention, concentration, and mental clarity, other proven benefits of mindfulness, such as empathy and emotional intelligence [13], could be explored in areas such as requirement elicitation. In particular, during interviews, requirements engineers should put themselves in the shoes of the client and the user in order to understand their needs with respect to the system to be developed. It is well known that the introverted character of the requirements engineer can sometimes make difficult to achieve the objectives during the interviews [14]. The impact of mindfulness in requirement negotiation or development team management could be also explored. Acknowledgements This work has been partially supported by the European Commission (FEDER) and the Spanish Government under projects HORATIO (RTI2018-101204-B-C21), APOLO (US-1264651), and EKIPMENTPLUS (P18-FR-2895). Competing Interest The authors declare that they have no known competing financial interests or personal relationships which have, or could be perceived to have, influenced the work reported in this article. 9 Information Classification: General References [1] P. Sedlmeier, C. Loße, and L. C. Quasten, “Psychological effects of meditation for healthy practitioners: an update”, Mindfulness, vol. 9, no. 2, pp. 371–387, 2018. [2] B. Bernárdez, A. Durán, J. A. Parejo, A. Ruiz-Cortés, “A controlled experiment to evaluate the effects of mindfulness in software engineering”, International Symposium on Empirical Software Engineering and Measurement (2014), pp. 1–10. [3] B. Bernárdez, A. Durán, J. A. Parejo, A. Ruiz-Cortés, “An experimental replication on the effect of the practice of mindfulness in conceptual modelling performance”, Journal of Systems and Software (2018), vol. 136, pp. 153–172. [4] B. Bernárdez, A. Durán, J. A. Parejo, N. Juristo, A. Ruiz-Cortés, “Effects of Mindfulness on Conceptual Modelling Performance: a Series of Experiments”, IEEE Transactions on Software Engineering. In press. [5] J. A. Parejo, S. Segura, P. Fernández, and A. Ruiz-Cortés, “Exemplar: An experimental information repository for software engineering research”, in Actas de las XIX Jornadas de Ingeniería del Software y Bases de Datos (JISBD’2014), 2014, pp. 155–159. [6] ACM Publications. “Artifact Review and Badging” [Online]. Available: https://bit.ly/3eFoARy [7] A. Brooks, J. Daly, J. Miller, M. Roper, M. Wood, “Replication of experimental results in software engineering”, Technical Report ISERN-96-10, University of Strathclyde, 1996. [8] M. Cruz, B. Bernárdez, A. Durán, J. A. Galindo, A. Ruiz-Cortés, “Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study from 2013 to 2018”, IEEE Access, vol. 8, pp. 26773– 26791, 2020. [9] D. T. Campbell and S. Julian, “Experimental and Quasi–Experimental Designs for Research”, United States: Wadsworth, 1963. [10] N. Shachtman, “Enlightenment Engineers: Meditation and Mindfulness in Silicon Valley”, San Francisco, CA, USA: WIRED Magazine, 2013. [11] Search Inside Yourself Leadership Institute, “Search inside your self–program impact report”, 2019. [Online]. Available: https://bit.ly/2LUMNdb [12] P. den Heijer, W. Koole, and C. J. Stettina, “Don’t forget to breathe: A controlled trial of mindfulness practices in agile project teams,” in Proc. Int. Conf. Agile Soft. Develop., 2017, pp. 103–118. [13] D. Goleman and R. J. Davidson, “Altered Traits: Science Reveals how Meditation Changes Your Mind, Brain, and Body”. Penguin, 2017. [14] L. F. Capretz. “Personality types in software engineering”. International Journal of Human– Computer Studies, 58 (2): pp. 207–214, 2003.