Project Management at University: Towards an Evaluation Process around Cooperative Learning

 Abstract—The enrollment in current Master's degree programs usually pursues gaining the expertise required in real-life workplaces. The experience we present here concerns the learning process of "Project Management Methodology (PMM)", around a cooperative/collaborative mechanism aimed at affording students measurable learning goals and providing the teacher with the ability of focusing on the weaknesses detected. We have designed a mixed summative/formative evaluation, which assures curriculum engage while enriches the comprehension of PMM key concepts. In this experience we converted the students into active actors in the evaluation process itself and we endowed ourselves as teachers with a flexible process in which along with qualifications (score), other attitudinal feedback arises. Despite the high level of self-affirmation on their discussion within the interactive assessment sessions, they ultimately have exhibited a great ability to review and correct the wrong reasoning when that was the case. Keywords—Cooperative-collaborative learning, educational management, formative-summative assessment, leadership training. I. INTRODUCTION RAINING graduates with competences that match the current demand for qualified work was appointed as a priority of the European Commission's "Education and Training 2020" strategy. Aside from the academic performance, it is important redesigning the interaction with students to better ensuring their professional growth and success. This is even more necessary as regards subjects around the leadership training requirements of novel educational management. This paper presents our experience in the application of a participative evaluation system for the subject "Technological Management II" in the Master of Telecommunication Engineering of the School of Engineering at the University of Seville in the 2016-7 academic year. The specific topic for this course is the Guide to the Fundamentals of Project Management PMBok v.5 [1], which is recognized worldwide as a fundamental reference in the application of knowledge and good project management practices. Aside from technical issues, the PMP application in a real-life scenario regards the knowledge of the rules (productivity, cooperation, service, money, deals) and further basic skills of understanding and analysing human, motivations behaviour. While in the private sector the professionals get across these Dr. Andrade-Pineda works at the Industrial Engineering and Management Science, University of Seville, Camino de los Descubrimientos s/n, 41092, Seville, Spain (corresponding author, e-mail: jandrade@ us.es). Dr. Leon, Dr Calle and Dr González-R work at the Industrial Engineering and Management Science, University of Seville, Camino de los Descubrimientos s/n, 41092, Seville, Spain. fundamental issues once they have deployed a certain career, teaching it from scratch in a European Credit Transfer System (ECTS) context necessarily needs to account for the particular starting point of our students in order to assure goals achievement. Along with the lack of a former professional experience, another handicap to face it the little time we have for teaching these PMP contents. Precisely, this has been the main motivation for designing a learning-centred programming of activities. The collaborative experience we report next turns around students in groups that analyze cases in a methodology that fosters skills related to teamwork, such as communication and collaborative problem solving skills, promoting exchange and collective construction of knowledge throughout the learning process [2]. The objective pursued is providing the students with a ground for the application of conceptual PMM contents in their workplaces. II. ALTERNATIVE ASSESSMENT In the educational context, assessment refers to the systematic process of gathering and discussing information to document the student-learning outcomes and to score the level of student achievement [3]. While in the past, knowledge transfer paradigm was the most common teaching practice, current efforts more and more aim at teaching centered on learning process and education based on the development of students’ competences. Instead of using the standard summative assessment for scoring the performance of students, this alternative assessment should provide more comprehensive insight into students’ achievements and gain authentic information about their knowledge, abilities, skills, attitudes, and competences developed during the teaching process. However, alternative assessment is still not adequately adopted in the educational system, mainly because of teachers’ lack of comfortability with the new paradigm [3]. According to [4], the new assessment paradigm necessarily encompasses new ways for planning the teaching and learning process. In the last years, certain experiences on formative assessment have been reported [5]-[9]. Reference [5] conducted a study at the University of Palermo based on asking students to formulate a series of questions (at different levels of difficulty) in areas determined by the teacher. From the set of questions that emerge, the students select a small subset which he first answers individually (first phase) and latter addresses within a small group (second phase). The experience ends when every group presents the answers to the other students in the class, and there is a discussion among the J. L. Andrade-Pineda, J.M. León-Blanco, M. Calle, P. L. González-R Project Management at University: Towards an Evaluation Process around Cooperative Learning T World Academy of Science, Engineering and Technology International Journal of Educational and Pedagogical Sciences Vol:13, No:1, 2019 32International Scholarly and Scientific Research & Innovation 13(1) 2019 ISNI:0000000091950263 Open Science Index, Educational and Pedagogical Sciences Vol:13, No:1, 2019 publications.waset.org/10009923/pdf groups, with the mediation and intervention of the teacher. Reference [6] assessed the cognitive, procedural and instrumental, and attitudinal competences, in the context of a degree subject at the School of Engineering of the University of Seville. The authors use the list of competences identified by [7]. In a first step, the students are requested to formulate questions regarding the specific subject (procedural and instrumental competences). Another student further evaluates the questions proposed. Once the response to the question posed by the students were collected, the authors measure not only the cognitive skills but also the extend of responsibility and critical thinking (each student assesses the correctness of answers provided by others). In [8] students have to ask questions on specific topics, through an e-learning platform which are then assessed according to its relevance, the correctness use of terminology, the level of difficulty and finally, the degree of multidisciplinary incorporated. In this experimental study, the authors found that the level of learning achieved by students who are questioned by other students in such a structured and interactive procedure is higher than when they only face teachers’ questionnaires. The self-regulation of learning and the positive interdependency among students is behind the experience reported by [9]. It consists of creating exam questions to medical students organized in small groups, motivating them with the compromise that 25% of the questions that appear in the final exam come from the bank of questions generated. The result is highly promising, showing an improved understanding of the required contents. All the above interactive experiences have a common feature: they have required of active students and of a teacher who gives motivation both individually and for the group. According to [10] such interactive methods can highly benefit the formative character of the evaluation processes. In what follows, we present how we planned the teaching and the assessment of the targeted PMM course on 2017. III. MOTIVATING STUDENTS A. Gamification In order to gain the motivation to learn, we have applied gamification. We have used the game in the early weeks of the term, around questions proceeding from real-life cases presented in class. Specifically, we have used a game-based classroom response system (Kahoot) to check students’ comprehension. Along the quiz-game sessions, we have promoted the critical understanding of the PMM practice, thereby going beyond the theoretical contents. The latter is very useful for young telecommunication engineers, which need to quickly adapt to the production on projects dynamics currently in use in the sector. In order to motivate an active role-play of students in the quiz-game, a fair reward was necessary. Each student cumulated points according to her/his Kahoot responses. The students are allowed to conform work teams, each one responsible for developing a PMM case-study whose score should be part of the final qualification for each individual; there is a list of case studies proposed by the teacher, which are picked up according to the bigger scores sum exhibited by the work teams. According to the self-determination theory [11], we can expect a positive response from our students because of: (i) the perceived autonomy in their actions, (ii) the winning spirit (leaderboard), and, (iii) the likely positive effect of the relaxed way teacher constructively analyzes their errors. B. Cooperation The need for ‘cooperating with others in an organized manner to attain a common goal’ is one of the first rules a telecommunication engineer learns when arrive at a real industrial environment. While the individual homework is considered in the ECTS context (one credit equivalent to 10h of teaching and 15-20h of work at home), how teamwork must be introduced is not clear at all. In our planning of the learning process, the students are required to organize in workgroups (according to their personal preferences) to thoroughly prepare a case study presentation. Once the students get persuaded that enriched case-based learning [12] arise from cooperating in an organized manner, they have been ready to address real-life case in workgroups. In our design, we have balanced the autonomy in guiding their individual efforts on PMP, with their participative activities together with the other workgroup members. Hence, the major issue is how to check the fair dedication of every member within the workgroup. In order to avoid solely a split of tasks (which is the natural way they would proceed), we have concerned each member in every workgroup in the knowledge of the whole project. To give raise to it, which member had to present the case in the qualification session has been known only 5 minutes before the event. Prior to it, every member is asked to score the active/passive role of the other members: 10 points to be spread among the rest of the teamwork in an anonymous manner. Observe that both things penalize the passive attitude or the lack of compromise on the workgroups. C. Roadmap for Performance Assessment Once the lecture period was finishing, our focus was pointed to the learning performance. Aiming to better diagnose the learning process and improve the maturation process of PMM concepts, we decided to apply a mixed summative-formative evaluation. In a classic summative assessment, along with the time commitment (dependent on the particular student's profile), the other key issue is the need for certifying the level of learning for each individual. On attempting to focus on the better accomplishment of the learning goals, different variants of evaluation are in use nowadays. In Spain, such efforts are typically based on what we call "Continuous Evaluation", a summative assessment which attempts to gain the student motivation by defining a path of progressive rewards as the course go on. However, this World Academy of Science, Engineering and Technology International Journal of Educational and Pedagogical Sciences Vol:13, No:1, 2019 33International Scholarly and Scientific Research & Innovation 13(1) 2019 ISNI:0000000091950263 Open Science Index, Educational and Pedagogical Sciences Vol:13, No:1, 2019 publications.waset.org/10009923/pdf strategy often leads to the undesired effect of demotivation once the student got the minimal qualification to pass the subject. Hence, it is not strange that disinterest emerges in the last weeks of class. Therefore, the teacher left a significant percentage of the final qualification to be function of a classic exam by the end of the lecture period. On the assumption that the students tend to focus their learning activities depending on the manner they will be evaluated, we have applied an interaction strategy for gaining their interest when most of lectures were taught. Persuaded of their necessity for being guided in their learning process, most students were open to this technique of revising the PMM contents. IV. M IXED S UMMATIVE -F ORMATIVE E VALUATION A. Aim and Scope We have conducted an experimental formative evaluation process on subject "Technological Management II", focused on identifying the needs for further work to improve student learning. We have decided applying the pragmatic concept of alternative assessment [4], thereby perceiving summative and formative assessment as complementary paradigms that are compatible and enrich each other. In short, we have supplemented the existing practice. B. Hypothesis Our first hypothesis is that, as Vigotsky claims "the student will learn more effectively when he does it in a cooperative way between peers" [13]. The teacher can be perceived as an expert in the matter, whose language or even the attitude when explaining is far of those of the students themselves. Thus, relating the knowledge that the student is acquiring with his own experience will make the learning more efficient. Our second hypothesis assumes the positive interdependence in the development of participatory sessions among the students in "Technological Management II". Similar to [9], we expect the improved understanding of the PMM contents from their participation in the selection of the questions to appear in a final exam. C. Procedure On the last three weeks of the course (end of May 2017), 18 students in the target subject are given a set of test questions with four possible answers (belonging to two levels of difficulty) in areas defined by the teacher. Among this wide number of questions, every student has to choose a subset of them according to certain instructions. Later, in a second stage these answers are transferred to a different student, who has to process whether the answers given by the first student are correct or not ... justifying its judgement. We have conducted this pilot experience along four sessions of 1.5 h each. Once achieved the two steps, the teacher has processed the material collected and provided the students with quick feedback (in next day). Furthermore, every feedback session was indeed a pedagogical forum, in which the students' interaction with each other allows them to go beyond the individual vision of the exam, thereby becoming a collaborative learning experience. According to [14], the procedure executed is a formative evaluation process since we: 1) Properly dose and regulate the pace of learning. 2) Give feedback of the effective learning, once processed the results from any evaluated activity. 3) Focus on the most valuable contents 4) Guide learning by effective techniques/methods 5) Report on the individual learning gaps/achievements. 6) Provide the roadmap for students. In Fig. 1 we present the instructions given for the earliest session. Observe the ‘If you are in doubt… don’t worry, just indicate it!. At the following lecture, we provide the students with the feedback of their answers. The best score was 45 correct answers out of 54, the average score 29 and the lowest score 23. As regards the number of judgement of others, the higher number was 10 and the average one, 5. Although the number of corrections is dependent on how the quality of the answer provide for the first student, a general differentiated level of attention is perceived between the first and the second step. Clearly, major attention is paid to the first stage, whereby students seem to be more engaged to the ‘exam’ with its name/surname than with that signed by a colleague. Fig. 1 First Session proposes solving 54 test questions in 45' The first session was indeed a training for the mechanism to rule the Student-Selected Questions formative sessions, to get a pool from which to pick a 40% of the test questions in the final exam up. In Fig. 2 we present the instructions for one of such sessions. On the collected data we clearly identify different motivational behaviors: from the one focus on selecting the easier questions, to the one which prefer facing the more challenging ones). We also observed that after World Academy of Science, Engineering and Technology International Journal of Educational and Pedagogical Sciences Vol:13, No:1, 2019 34International Scholarly and Scientific Research & Innovation 13(1) 2019 ISNI:0000000091950263 Open Science Index, Educational and Pedagogical Sciences Vol:13, No:1, 2019 publications.waset.org/10009923/pdf several applications of this mechanism, the students are concerned by their own reputation: they take more care on their responses. Hardly ever the students are right in all their answers, since the degree of difficulty was high. Nevertheless, the general observation is that they were conservative and provide soft corrections on their judgement of their partners’ questionnaires. Conversely, on the participative sessions they are more relaxed and participate actively in the discussions around the contents that needed to be reinforced for the correct learning. Fig. 2 Second Session to get the Student-Selected Questions D. Outcomes The assessment of the learning outcomes from the discussed procedure has been done according to Kirkpatrick’s model [15], with its four levels of evaluation: reaction, learning, behavior, and result. At the first two levels, the trainee evaluation is dominant in relevance, whereas the student evaluation is dominant in relevance at the last two levels [16]. At the reaction level, we have measured that the degree of satisfaction was high concerning: (i) certainty in the concepts discussed (the participation climate was crucial on making student lose the fear to ask) around real case studies and, (ii) the delay in the clarification of doubts was shortened (hardly 2 days after they have had the collaborative sessions). At the learning level, we provided the feedback in dynamic and participative mode. The focus was on the clarification of concepts, far from a retentive concept of learning. The feedback sessions distinguished the in-deep discussion of the more confusing concepts. Furthermore, they were mostly connected to the case studies they were going to exposure in the following weeks. It is noteworthy that we have observed that the initial utilitarian motivation for participating –i.e. the instrumental promise of transferring questions to the final test exam-, was turned into a deeper interest on the PMM concepts. As we pretend, a clear positive interdependence appeared in the development of the participatory sessions. The discussion of doubts and correctness/error of their interpretation of test questions, have pointed to the common good of PMM contents apprehension meanwhile every student clarifies their practical application to real-life scenarios. V. C ONCLUSION There is a clear need of experimenting with new interaction strategies for the quality of academic motivation. The conclusions after the piloted experience for teaching PMM are two-fold. First, that the use of game-based tools is really easy to incorporate to the existing teaching practice, and are a great opportunity to motivate students in following the teaching lectures. The second conclusion is that the mixed summative-formative evaluation presented here is so timeconsuming that is affordable only in case of a reduce number of students. In order to be effective, the teacher needs to provide the feedback for the collaborative sessions in short time. Our experimental study relied on a small group (<20), and the alternative assessment reported has served the purpose of encouraging the analytic and critical thinking skills of students. Furthermore, we have changed their focus from technology to management, as the main driver to turn innovations into real products. It is typical in young engineers to think that what provide value to engineering solutions was the technological innovations inside, whereas value is only attained when in addition we are able to put them on effective exploitation. R EFERENCES [1] Project Management Institute “A guide to the project management body of knowledge (PMBOK® Guide) – Fifth Edition, ANSI/PMI 99-001- 2013. [2] Saldaña, O., Rodríguez-Carballeira, Á., Espelt, E., Jiménez, Y., Porrúa- García, C., Escartín, J., Vidal, T. “Development of psychosocial case studies by students to improve their ability to understand and analyze human behavior” in Proceedings of ICERI 2015 Conference2015 Conference, pp. 1097–1102 [3] Stiggins, R. “Student-centered classroom assessment”, New York, Prentice Hall, 1997. [4] Letina, A. “Application of Traditional and Alternative Assessment in Science and Social Studies Teaching”, Croatian Journal of Education, 17(1), pp.137–152., 2015. [5] Ciancimino, E., Canella, S. “The Alternative Assessment Experiment”, The International Journal of Learning, 15(2), pp. 295–300, 2008. [6] González-R, PL, Framiñán, JM, León Blanco, JM, Pérez P., “Experiencias en la aplicación de un nuevo sistema de Evaluación Formativa en el ámbito del EEES” in International Conference on Industrial Engineering and Industrial Management, pp. 205–206, 2009. [7] Sadler, D.R. “Formative assessment and the design of instructional systems”, Instructional Science, 18, pp. 119–144, 1989. [8] Cannella, S., Ciancimino, E., Campos, M. L. “Mixed e-assessment: An application of the student-generated question technique” in IEEE Education Engineering Conference, EDUCON 2010, pp. 769–773, 2010. [9] Papinczak, T., Peterson, R., Babri, A. S., Ward, K., Kippers, V., Wilkinson, D. “Using student-generated questions for student-centred assessment”, Assessment & Evaluation in Higher Education, 37(4), pp. 439–452, 2012. World Academy of Science, Engineering and Technology International Journal of Educational and Pedagogical Sciences Vol:13, No:1, 2019 35International Scholarly and Scientific Research & Innovation 13(1) 2019 ISNI:0000000091950263 Open Science Index, Educational and Pedagogical Sciences Vol:13, No:1, 2019 publications.waset.org/10009923/pdf [10] Bergman, B. “Interactive assessment – A new element in the examination process” in 8th QMOD 2005 Proceedings, 2005. [11] Ryan, R. M., Deci, E. L. “Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being“, American psychologist, 55(1), pp. 68–78, 2000. [12] Barkley, E. F., Cross, K. P., Major, C. H. “Collaborative learning techniques: A handbook for college faculty”, San Francisco, Jossey- Bass, 2005. [13] Yasnitsky, A. “Vygotsky circle as a personal network of scholars: Restoring connections between people and ideas”, Integrative Psychological and Behavioral Sciences, 45, pp. 422–457, 2011. [14] Jorba, J.; Sanmartí, N. “La función pedagógica de la evaluación. Evaluación como ayuda al aprendizaje”. Madrid, Editorial Grao, 2000. [15] Kirkpatrick, D. “Revisiting Kirkpatrick’s four-level model”. Training and Development, 50(1), pp.50–54, 1996. [16] Moldovan, L. “Training Outcome Evaluation Model”, Procedia Technology, 22, pp. 1184–1190, 2016. José L. Andrade-Pineda is a Postdoctoral Researcher at the Robotic, Vision and Control Group at the University of Seville. He is trained as a Telecommunication Engineer (2000), with seven years of experience on the Project Management of IT solutions. Since 2007, he is focused on Operations Research, area in which became PhD (2015). He is currently researching on Transport and Logistic Problems (Canca, D., Barrena, E., De los Santos, A., Andrade-Pineda, J.L.: “Setting lines frequency and capacity in dense railway rapid transit networks with simultaneous passenger assignment”. Transportation Research. Part B: Methodological. 2016. Vol. 93, pp. 251-267), lately around the use of drones (Calle, M, Andrade-Pineda, J.L., Gonzalez-R., P.L. Leon-Blanco, J.M., Canca, D.: “A Tandem Drone-ground Vehicle for Accessing Isolated Locations for First Aid Emergency Response in Case of Disaster”. International Joint Conference on Computational Intelligence, IJCCI 2018). He is also a teacher at the School of Engineering, University of Seville. He is specialized in Innovation and Technology Transference processes. Currently involved in the teaching of procedural subject required for the quick set-up of post-graduate students and their insertion to the industrial context. Jose Miguel Leon-Blanco is an Industrial Engineer (1998) and a PhD (2009) in the University of Seville, Spain. His doctoral thesis dealt with the use and tuning of parallel algorithms applied to the solution of task scheduling problems. He is a PhD Lecturer at the School of Engineering, University of Seville, Spain since 2001. His main teaching areas are computer science and business administration. His research interests are high performance computing applied to industrial engineering problems. He has also been working on the applications of neural networks to the design of pharmaceutical drugs (Leon- Blanco, J.M., Gonzalez-R., P.L., Calle, M., Canca, J.D., Rabasco, A.M., “Artificial Neural Networks as Alternative Tool for Minimizing Error Predictions in Manufacturing Ultradeformable Nanoliposome Formulations”. Drug Development And Industrial Pharmacy. 2018. Vol. 44.(1) pp. 135-143) and in the field of education (Leon-Blanco, J.M., González-R., P.L., Andrade- Pineda, J.L. “Motivación a la Inversion Mediante un Juego de Bolsa”. I Congreso Internacional de Innovación en Tendencias Educativas. Facultad de Educación, Universidad de Sevilla. 2017). Marcos Calle is an Associate Professor at the School of Engineering in the University of Seville, Spain. He has been teaching the last ten years, mainly on matters regarding to Business and Systems Modelling. He is also interested on Transference Processes, area where he started his professional career (three years working in a technological center named Instituto Andaluz de Tecnología, IAT). His research interests are Production and Business Simulation and Modeling, with recent works as: (i) González-R, P.L, Calle, M.: Chapter 4: “Impact of Reworking in Job-Shop Environments Controlled by Kanban System”. pp. 63-83. In: Production Management: Advanced Models, Tools, and Applications for Pull Systems. CRC PRESS. TAYLOR & FRANCIS Group. 2017. ISBN 9781138032.; and (ii) Calle, M.; González-R, P.L.; Leon, J.M.; Pierreval, H.; Canca, D. (2016) “Integrated management of inventory and production systems based on floating decoupling point and real-time information: A simulation based analysis”. International Journal of Production Economics, 181, pp. 48–57. He is also an easy-going IT developer which applies innovative teaching approaches as Gamification and Collaborative Learning within his lectures (Calle, M., León Blanco, J.M., Andrade-Pineda, J.L. (2017) “Caso práctico de la ludificación en la enseñanza universitaria de marketing y dirección estratégica”. INNTED 2017). Pedro Luis Gonzalez-R is trained as BSc Electrical Engineer and later as MSc Industrial Engineer and PhD. He is an Associate Professor at the School of Engineering in the University of Seville, Spain. He has been a researcher in several research projects and published papers in international refereed journals (Gonzalez-R., P.L., Calle, M., Andrade-Pineda, J.L. “Job shop management of products under internal lifespan and external due date”. International Journal of Production Research. 2018. pp. 1-18.) or (Andrade-Pineda, J.L., Canca, D., Gonzalez-R., P.L. “On modelling non-linear quantity discounts in a supplier selection problem by mixed linear integer optimization”. Annals of Operation Research. 2017. Vol. 258 (2). pp. 301-346). He has enjoyed several stays at foreign research centers. He is evaluator for several Teaching Innovation agencies. His research interests include information systems, optimization by operations research methods and heuristics optimization. He is also interested in teaching and education, design, modelling, simulation and optimization of manufacturing systems. World Academy of Science, Engineering and Technology International Journal of Educational and Pedagogical Sciences Vol:13, No:1, 2019 36International Scholarly and Scientific Research & Innovation 13(1) 2019 ISNI:0000000091950263 Open Science Index, Educational and Pedagogical Sciences Vol:13, No:1, 2019 publications.waset.org/10009923/pdf