A data-driven approach for a project management methodology for R&D Projects.

Adierazpen Grafikoa eta Ingeniaritzako Proiektuak Saila Departamento de Expresi´on Gr´afica y Proyectos de Ingenier´ıa A data-driven approach for a project management methodology for R&D Projects by Leonardo Sastoque Pinilla Supervised by Nerea Toledo Gandarias and Norberto L´opez de Lacalle Dissertation submitted to the Department of Graphic Design and Engineering Projects of the University of the Basque Country (UPV/EHU) as partial fulfilment of the requirements for the PhD degree in Project Engineering Bilbao, December 2022 (cc)2023 EDWAR LEONARDO SASTOQUE PINILLA (cc by-nc-nd 4.0) ”Caminante, son tus huellas el camino, y nada m´as; Caminante, no hay camino: se hace camino al andar. Al andar se hace el camino, y al volver la vista atr´as se ve la senda que nunca se ha de volver a pisar, Caminante, no hay camino, sino estelas en la mar.” Antonio Machado III Acknowledgments. Gloria, I still remember the night it all happened, the music playing in the background on one of the few electrical appliances we had. Sitting on the floor by the window and hugging each other, while watching how Bogot´a was giving us one of its magical nights, where the cloud of pollution gave way to a starry night and a red moon at dusk. 1 I remember your story, your dreams, your question; I remember every single second that passed while I was thinking, even though I had already made up my mind from the moment your story began. I remember the emotion, I remember the fear, I remember the love, I will never forget the love, and at this moment, I can’t help but feel proud of you, of me and of the decision we made because of that moment on our life changed completely. So that gift you gave me changed my whole life 2. I wonder if it was bravery or stupidity, and I still cannot answer myself. You know that I don’t consider myself a particularly brave or notoriously stupid person (at least according to the definition of the great Carlo M. Cipolla [1]. Despite the tireless effort I make night after night to prove you otherwise; we have been able to walk this path together. Hard, and as hard as it can be, challenging day after day, complex as we would never have thought, but exciting beyond compare. That is why the satisfaction it gives me to finish this road and to be able to move forward to set new goals, new places, new learning, new paths, and new lives. That is why I thank you, Mamor, especially you, but also all the people who were there in one way or another. Tatiana Pineda, my tutor, friend, advisor and a big part of my everything. Sara Sendino, for her company and guidance from day one. Unai L´opez-Novoa, for all the timely advice and guidance in the moments when I needed it the most. Santiago G´alvis, ”Mi Perrito”, for the closeness and remembrance. Nargiza Mikhridinova, of course, the example, the dedication and her openness that I needed to learn so much. Bertha Ngereja who is the most positive human being I have ever had the pleasure of meeting. Carsten Wolff, one of the most brilliant human beings I have ever met, for his teachings, support and kindness. I˜nigo Santisteban, the best friend we could have wished for. Itziar Garrote and Octavio Pereira, for their example, education, conversations, tenacity and time. Aitor Irazabal, Jon Ander Iturrioz and Olatz Parola, without whom, without a doubt, the day-to-day life would have been much harder to bear. Special mention should be made of my thesis directors, Norberto L´opez de Lacalle and Nerea Toledo Gandarias, without whom none of what I have written here, nor the dreams I have built, would have been founded. Berti, for trusting me from the first moment, even I was unable to do so, for supporting me at every opportunity, for acting as a mentor, and for his teachings. Immeasurable. Thank you. 1Cerati, G., Bosio, Z. (1992). Luna Roja - Dynamo [CD]. Argentina: Sony Music. 2Jorge Drexler (2017). Pongamos que hablo de Mart´ınez - Salvavidas de hielo [CD]. Espa˜na: Warner Music IV Thanks also to every member of the CFAA, a place that has been my second home for the last five years, where I have been allowed to learn, make mistakes, pick myself up whenever I need to, and feel an essential part of it. Finally, I have always thought that one never walks alone, but that every step we take with us, carrying with us each and every one of the people who have been part of our history. So, of course, my family in Colombia, my parents, brothers, sisters, aunts, cousins, and friends. I carry each and every one of you with me every step of the way, and I am what I am because of what you allowed me to learn from all of you. I also thank those I consider my friends from the bus stop: Andr´es, Nagore, Esti, Javi, Javier, and Silvia, who have allowed us to feel part of each other and to raise our daughter in a community. And you are still too young to read and understand this, Maite, but in time I hope you will know that this is also for you; of course, it is for you. You are my life, love, and desire to live and move forward, achieve goals, and improve myself daily. So THANK YOU, with capital letters, because my heart is completely whole for you. Because I have felt things that I didn’t even know existed and didn’t think I deserved, for filling me with pride every day for being your father and feeling enormously grateful for having the privilege of seeing you grow day by day. I love you beyond measure. To all of you and those I have unfortunately not mentioned here, as the greatest said, ”Gracias, totales”. V Gloria, a´un recuerdo la noche en que todo ocurri´o, la m´usica sonando de fondo en uno de los pocos electrodom´esticos que ten´ıamos, sentados en el suelo junto a la ventana y abrazados viendo como Bogot´a nos regalaba una de sus noches m´agicas, donde al anochecer, la nube de contaminaci´on daba paso a una noche estrellada y a una luna roja 3. Recuerdo tu historia, recuerdo tus sue˜nos, recuerdo tu pregunta, recuerdo cada uno de los segundos que pasaron mientras pensaba; aunque ya me hab´ıa decidido desde el momento en que comenz´o tu narraci´on. Recuerdo la emoci´on, recuerdo el miedo, recuerdo el amor, nunca olvidar´e el amor, y en este momento no puedo evitar sentirme orgulloso de ti, de m´ı y de la decisi´on que tomamos, porque desde ese momento nuestra vida cambi´o por completo. Ese regalo que me diste cambi´o mi vida entera 4. Me pregunto si fue valent´ıa o estupidez, y todav´ıa no soy capaz de responderme. Sabes que no me considero una persona especialmente valiente ni notoriamente est´upida (al menos seg´un la definici´on del gran Carlo M. Cipolla [1], y a pesar del esfuerzo inagotable que hago noche tras noche para demostrarte lo contrario); pero gracias a ello hemos podido recorrer juntos este camino. Duro, y tan duro como puede ser, desafiante d´ıa tras d´ıa, complejo como nunca hubi´eramos pensado, pero emocionante sin comparaci´on. Por eso, la satisfacci´on que me da terminar este camino y poder avanzar para plantear nuevas metas, nuevos lugares, nuevos aprendizajes, nuevos caminos, nuevas vidas. Por eso te doy las gracias. mamor, especialmente a t´ı, pero tambi´en a todas las personas que estuvieron ah´ı de una u otra manera. Tatiana Pineda, mi tutora, mi amiga, mi consejera y una gran parte de mi todo. Sara Sendino, por su compa˜n´ıa y gu´ıa desde el primer d´ıa. Unai L´opez-Novoa, por todos los consejos oportunos y la orientaci´on en los momentos en que m´as lo necesitaba. Santiago G´alvis, ”Mi perrito”, por la cercan´ıa y el recuerdo. Nargiza Mikhridinova, por supuesto, el ejemplo, la dedicaci´on y su franqueza que tanto necesito aprender. Bertha Ngereja, el ser humano m´as positivo que he tenido el placer de conocer en la vida. Carsten Wolff, uno de los seres humanos m´as brillantes que he conocido en toda mi vida, por sus ense˜nanzas, su apoyo y su amabilidad. I˜nigo Santisteban, el mejor amigo que hemos podido desear. A Itziar Garrote y a Octavio Pereira, por su ejemplo, ense˜nanzas, conversaciones, tes´on y tiempo. A Aitor Irazabal, Jon Ander Iturrioz y Olatz Parola, sin los cuales, sin duda, el d´ıa a d´ıa hubiera sido mucho m´as dif´ıcil de aguantar. Menci´on especial merecen mis directores de tesis, Norberto L´opez de Lacalle y Nerea Toledo Gandarias, sin los cuales nada de lo que he escrito aqu´ı, ni los sue˜nos que he construido, habr´ıan tenido fundamento. Berti, por confiar en m´ı desde el primer momento cuando ni siquiera yo era capaz de hacerlo, por apoyarme en cada ocasi´on, por ejercer de mentor, por las ense˜nanzas. Inconmensurable. Gracias. Gracias tambi´en a todos y cada uno de los miembros de la CFAA, un lugar que ha sido mi segundo hogar durante los ´ultimos 5 a˜nos, donde me han permitido aprender, 3Cerati, G., Bosio, Z. (1992). Luna Roja - Dynamo [CD]. Argentina: Sony Music 4Jorge Drexler (2017). Pongamos que hablo de Mart´ınez - Salvavidas de hielo [CD]. Espa˜na: Warner Music VI equivocarme y levantarme cada vez que lo necesitaba, y sentirme una pieza importante de all´ı. Por ´ultimo, siempre he pensado que uno nunca camina solo, sino que cada paso que damos lo hacemos llevando con nosotros a todas y cada una de las personas que han formado parte de nuestra historia. As´ı que, por supuesto, mi familia en Colombia, mis padres, mis hermanos y hermanas, mis t´ıas, mis primos, mis amigos. A todos y cada uno de ustedes los llevo conmigo en cada paso del camino, y soy lo que soy gracias a lo que me permitieron aprender de todos ustedes. Tambi´en agradezco a los que considero mis amigos de la parada del autob´us, Andr´es, Nagore, Esti, Javi, Javier, Silvia, que nos han permitido sentirnos parte y criar a nuestra hija en comunidad. Y t´u eres a´un demasiado joven para leer y entender estas palabras, Maite, pero con el tiempo espero que sepas que esto tambi´en es para ti, por supuesto que es para ti. T´u eres mi vida, mi amor, mis ganas de vivir y avanzar, de alcanzar metas y de superarme cada d´ıa. As´ı que GRACIAS, con may´usculas, porque por ti mi coraz´on est´a completamente lleno, porque he sentido cosas que ni siquiera sab´ıa que exist´ıan y que no cre´ıa merecer, por llenarme de orgullo cada d´ıa por ser tu padre y sentirme enormemente agradecido por tener el privilegio de verte crecer d´ıa a d´ıa. Te quiero sin medida. A todos ustedes y a los que lamentablemente no he mencionado aqu´ı, como dijo el m´as grande, ”Gracias, totales”. VII Abstract Research and Development (R&D) projects are fraught with significant problems, such as the likelihood of failure, the high rate of projects ending without results, the changing project scope, the prolonged project life cycle, and the clash between the interests of academics and companies. Furthermore, R&D projects are also characterised by the difficulty of bounding to defined periods and planning. The non-fixed scope of these projects can change due to internal and external factors. Besides that, the Technology Readiness Level (TRL) in which the R&D project is conducted determine its characteristics and challenges. Furthermore, the quality of the result of an R&D project is seen only at the end of it. This result is formed by the progressive and cumulative realisation of the activities that make it up. It also depends on several features, characteristics and attributes that contribute to meeting the needs and expectations of the stakeholders. The main goal of this thesis is to overcome some of the issues inherent to these types of projects, developing a project management methodology based on Earned Quality Method (EQM) and data analysis to improve the efficiency of R&D projects in a nearreal production environment in a TRL 5-7. The thesis relies upon published papers that propose measuring and improving the management of Research and Development (R&D) projects. The method leans on the formulation and gradual and recurrent evaluation of quality criteria as a performance indicator of the work carried out. The way to develop the idea stands on the concept that quality is a measurable quantity that accumulates throughout the project. The proposed project management methodology is built on three main aspects: Collaboration between University and Industry The correct interpretation of the TRL where research projects are developed The study of different metrics for project management, such as the measurement of the success of projects, the Key Performance Indicators (KPIs) of a project-based organisation, and the EQM EQM is analysed, used and taken a step further by applying it to R&D projects and proposing new contributions for the definition of quality criteria, a holistic view of the method, the reinforcement of EQM, and a series of recommendations for its correct implementation. The methodology has been tested with three actual use cases with different characteristics in terms of project size, funding and team members; and validated on an R&D Centre in Advanced Manufacturing in Aeronautics. The pillars of the thesis are focused on the analysis of the mentioned components and their integration for the development of a methodology to improve the efficiency in the use of resources and quality of obtained results in the R&D projects’ framework. The results have been presented in four publications at academic conferences and three original papers submitted to scientific journals of the JCR’ quartile one and two 5, and a final one in the final process of preparation. The key findings of these studies demonstrate 5Scopus Sources - https://www.scopus.com/sources.uri - Last Access: 20/12/2022 XV List of abbreviations. •ACWP: Actual Contribution of the Work Performed. •AM: Additive Manufacturing. •CCS: Critical Chain Scheduling. •CFAA: Centro de Fabricaci´on Avanzada Aeron´autica (Advanced Manufacturing Centre for Aeronautics: In English). •CNC: Computer Numerical Control. •CP: Control Points. •CPM: Critical Path Method. •CSC: Critical Success Criteria. •CSF: Critical Success Factor. •CTQs: Critical To Quality. •DEA: Data Envelopment Analysis. •DMAIC: Define, Measure, Analyse, Improve, Control. •EC: European Commission. •EQM: Earned Quality Method. •EQWP: Earned Quality of the Work Performed. •EU: European Union. •EVM: Earned Value Management. •EVMS: Earned Value Management System. •FoF: Factories-of-the-Future. •FRED: First Requirements Elucidator Demonstration. •IoT: Internet of Things. •KPIs: Key Performance Indicators. •KTTOs: Knowledge and Technology Transfer Organisations. •MPCS: Multidimensional Project Control System. •MRL: Manufacturing Readiness Level. •NASA: National Administration of Space Agency. •OEM: Original Equipment Manufacturer. •PBEV: Performance-Based Earned Value. •PCTI: Science, Technology and Innovation Plan (in Spanish). •PCWP: Planned Contribution of the Work Performed. •PCWS: Planned Contribution of the Work Scheduled. •PERT: Program Evaluation and Review Technique. •PLC: Programmable Logic Controller. XVI •PMO: Project Management Office. •PMs: Project Managers. •PPP: Phased Project Planning. •PQWP: Planned Quality of the Work Performed. •PQWS: Planned Quality of the Work Scheduled. •QBS: Quality Breakdown Structure. •QC: Quality Criteria. •QPI: Quality Performance Index. •QV: Quality Variance. •R&D: Research and Development. •SLR: Systematic Literature Review. •SMEs: Small and Medium Enterprises. •TRL: Technology Readiness Level. •TTOs: Technology Transfer Offices. •UIC: University-Industry Collaboration. •VoB: Voice of the Business. •VoC: Voice of the Customer. •WBS: Work Breakdown Structure. Contents Acknowledgments ...................................................... VI Abstract. ...............................................................VIII Resumen. ...............................................................XIII List of abbreviations ....................................................XVI Part I Description of Contributions. 1 Introduction......................................................... 3 1.1 Presentation. ..................................................... 3 1.2 Motivation........................................................ 4 1.2.1 Personal motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Theoretical framework and methodology. ........................... 11 2.1 Introduction. ..................................................... 11 2.2 Theoretical Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.1 University-Industry collaboration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.2 Technology Readiness Level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.3 Project management metrics and Key Performance Indicators for R&Dprojects. .............................................. 28 2.2.4 R&D Project Management Methodology. . . . . . . . . . . . . . . . . . . . . . . . 39 2.3 Research methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3 Hypothesis and Objectives. ......................................... 51 3.1 Hypothesis........................................................ 51 3.2 Objectives........................................................ 53 3.2.1 General Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.2.2 Secondary Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 XVIII Contents 4 Summary and discussion of the results. ............................. 55 4.1 Summary and discussion.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.1.1 First Publication CP.1 - Conference Paper: ”Patterns for international cooperation between innovation clusters. Cases of CFAA and ruhrvalley”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.1.2 Second Publication CP.2 - Conference Paper: ”Building Cooperation between Innovation Clusters Based on Competences Requirements. Case of CFAA and ruhrvalley”.. . . . . . . . . . . . . . . . . . . 58 4.1.3 Third Publication JP.1 - Original Journal Paper: ”TRLs 5–7 advanced manufacturing centres, practical model to boost technology transfer in manufacturing”. . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.1.4 Fourth Publication CP.3 - Conference Paper: ”Assessing the success of R&D projects and innovation projects through project management life cycle”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.1.5 Fifth Publication JP.2 - Original Journal Paper: ”Project Success Criteria Evaluation for a Project-Based Organization and Its Stakeholders - A Q-Methodology Approach”. . . . . . . . . . . . . . . . . . . . . 62 4.1.6 Sixth Publication JP.3 - Original Journal Paper: ”Identification of key performance indicators in project-based organisations through theleanapproach”........................................... 64 4.1.7 Seventh Publication CP.4 - Conference Paper: ”Hybrid Project Management Methodology for R&D, Innovation and R&D&I ProjectsinCFAA”........................................... 65 4.1.8 Eighth Publication JP.4 - Original Journal Paper: ”A data-driven approach for a new project management methodology based on qualityincrements”. ......................................... 66 References ..............................................................115 Part II Conclusions 5 Conclusions and future research. ....................................129 5.1 Conclusions.......................................................129 5.2 Futureresearch....................................................130 Part III Appendix 6 University - Industry Collaborations. ...............................137 6.1 CP. 1 - Conference Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 6.2 CP. 2 - Conference Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 7 Technology Readiness Level. ........................................151 7.1 JP 1 - Original Journal Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Contents XIX 8 Project Management Metrics........................................167 8.1 CP. 3 - Conference Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.2 JP. 2 - Original Journal Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 8.3 JP. 3 - Original Journal Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 9 Project Management methodology for R&D projects. ...............217 9.1 CP. 4 - Conference Paper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 9.2 JP. 4 - Original Journal Paper (In process).. . . . . . . . . . . . . . . . . . . . . . . . . . . 227 10 Communications and other Publications. ............................229 10.1 Press Communications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 10.2Otherpublications.................................................230 11 Data Appendix. .....................................................231 11.1CFAA’sKPIs. ....................................................231 11.2 Proposed databases and classification scheme.. . . . . . . . . . . . . . . . . . . . . . . . . 238 11.3 Critical Success Criteria as in Sastoque et al. [10] . . . . . . . . . . . . . . . . . . . . . 241 List of Figures 2.1 Drivers for university third mission [14]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 TRL and phases of development, based on [15]. . . . . . . . . . . . . . . . . . . . . . . . 24 2.3 Conceptual model of the research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.4 Collectingmethods ................................................ 45 2.5 First Publication: ”Patterns for International Cooperation between Innovation Clusters. Cases of CFAA and ruhrvalley” - Research Methodology...................................................... 46 2.6 Second Publication: ”Building Cooperation between Innovation Clusters Based on Competences Requirements. Case of CFAA and ruhrvalley.” - ResearchMethodology ............................................. 46 2.7 Fifth Publication: ”Project Success Criteria Evaluation for a ProjectBased organisation and Its Stakeholders—A Q-Methodology Approach.” -ResearchMethodology............................................ 48 2.8 Research Methodology JP4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.1 ResearchOnion ................................................... 56 4.2 Proposed Project Management Methodology in phases and activities . . . . . 72 4.3 Proposed Project management methodology including activities and documents........................................................ 74 4.4 QBS-WBSExample.............................................. 81 4.5 QBS-WBSProject3 .............................................101 List of Tables 2.1 Original TRL model [16] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.1 CSC and KPIs chosen per project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.2 Quality Criteria definition - Project 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.3 Quality Criteria definition - Project 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.4 Quality Criteria definition - Project 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.5 Allocation scheme of the potential contribution of the activities to quality 100 4.6 Phase 1: QBS-WBS levels 2, 3 and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.7 Phase 2: QBS-WBS levels 2, 3 and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.8 Phase 3: QBS-WBS levels 2, 3 and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.9 Phase 4: QBS-WBS levels 2, 3 and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.10 Phase 5: QBS-WBS levels 2, 3 and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.11 Phase 6: QBS-WBS levels 2, 3 and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.12 Gantt Chart of the project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.13 PCWS:P3Ph1....................................................106 4.14 PCWS:P3Ph2....................................................106 4.15 PCWS:P3Ph3....................................................106 4.16 PCWS:P3Ph4....................................................107 4.17 PCWS:P3Ph5....................................................107 4.18 PCWS:P3Ph6....................................................107 4.19 Planned quality of work scheduled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.20 Planned contribution of the work performed . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.21 Actual contribution of work performed - Phase 1 . . . . . . . . . . . . . . . . . . . . . . 109 4.22 ACWP-P3Ph1...................................................110 4.23 ACWP-P3Ph2...................................................110 4.24 ACWP-P3Ph3...................................................110 4.25 ACWP-P3Ph4...................................................111 4.26 ACWP-P3Ph5...................................................111 4.27 ACWP-P3Ph6...................................................111 4.28 Earned quality of work performed - Project 3 . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.29 Quality variance - Project 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6 CHAPTER 1. INTRODUCTION. new technology can be challenging and hazardous. This approach necessitates the development of disruptive practices. One of the core benefits of this collaboration can be seen by more researchers and company staff working with KTTOs, sharing experiences, points of view, and solutions to problems encountered during the execution of R&D projects. Several ways to measure the success of this collaboration can be set [26]. However, the most important is not just how many projects are being carried out but the success rate of those projects aligned to the organisation’s Key Performance Indicators (KPIs). Alternatively, if it has adequately solved the problem, fit with the gap observed, or fulfilled the expectations and needs of the different stakeholders. Limited access to public or private funding, a long payback period for some types of products or wrong market research can lead to the loss of efforts from many years in developing and researching new products. This phenomenon is intricate for large companies and Small and Medium Enterprises (SMEs). As a result, many ideas and discoveries developed at early TRLs remain in universities, with researchers unable to commercially capitalise on their innovations due to a lack of entrepreneurial skills, business know-how and contacts needed to gain access to the business world [25]. However, these intermediate KTTOs are fundamental and essential instruments within the innovation process to be dismissed. Fortunately, a new way of managing companies, projects, and universities based on data is emerging thanks to Industry 4.0. In this new era, large amounts of data are available within companies, universities, and KTTOs; and different types of data analysis techniques rise to maximise the benefit from this data, allowing to develop of new processes, tools, practices, and theories in project management to improve the control of projects, even in harsh environments such as KTTOs or an R&D environment. Given these conditions, quality assurance is essential during the project life cycle (from the customer’s and project team’s approach). This is why, based on the premise that quality is a quantifiable quantity and following the client’s and project team’s expectations, this thesis is proposed to analyse measures to control and assess the project’s quality performance throughout its life cycle. It also highlights how the use of information to establish significant quality deviations is intended to help Project Managers (PMs) make timely choices to rectify the project’s path. The most important concepts for the thesis are the following ones: 1. Collaboration between public and private organisations for the growth of R&D in local environments through a quick technology transfer; 2. the identification and measurement of success criteria for R&D projects for both PMs and stakeholders; 1.2. MOTIVATION. 7 3. identification of KPIs for project-based organisations based on lean principles through a data-driven approach; 4. the identification of quality criteria for an R&D project based on success factors, organisational KPIs and client-defined criteria; based on a data-driven approach to control the quality of the work performed and the quality of the delivered product; 5. the need to break down quality criteria into small pieces to control the outcome of a project; 6. the development of a project management methodology based on the concept that quality is measurable and achievable throughout the project life cycle and on generating knowledge to improve project performance. These are some reasons why the need for control R&D projects is increased in times as turbulent as the ones we live in. Looking for ways to improve the efficiency of R&D investments given to public-private entities, achieving quality results of the projects and being able to take decisions in time to avoid further R&D losses are some of the strategies to be followed to limit the impact of the current economic context. Encouraged by the current socio-economic scenario, we set a general objective to create a way to carry out these strategies and thus improve the efficiency of R&D projects in a quasi-real production environment in a TRL 5 - 7, which will be described in Section 4.1.8. This general objective is complemented through different types of scientific communications with the following specific objectives: 1. To define the collaboration strategies between public-private entities through the study of related innovation centres whose objective is to promote scientific, economic, and social growth in their regions through collaboration with the public and private sectors; 2. to analyse the role of the TRL in technology transfer and how it affects regions with such centres; 3. to analyse different metrics of R&D projects, such as the success of projects and what it means for stakeholders, the identification of KPIs for organisations in charge of developing such projects and the measurement of project progress through quality metrics; 4. to develop an R&D project management methodology based on data analysis to improve the management of this type of project. The scientific papers presented in this thesis summarise the experience and knowledge acquired on a theoretical and practical level in the management of R&D projects in public-private centres in collaboration with industry and located in TRL 5 - 7, from the beginning of the TRL analysis in September 2019 to the definition of the methodology in December 2022. 8 CHAPTER 1. INTRODUCTION. Intending to describe the fulfilment of the proposed objectives of this thesis, we begin with an analysis of the current state of research centres like the Advanced Manufacturing Centre for Aeronautics - CFAA (in Spanish) 3) from a project management point of view. Thanks to a bench-marking study, a general analysis of the current state of the centres and some general recommendations for the Basque Country scenario. The research was developed to understand and go deeper within the production chain and the TRL of the CFAA. A research paper has been published in this Section [5]. After that, for the methodology, it was necessary to analyse and study the role of university-industry collaboration to complement the previous point. Therefore, two research projects were developed: i) The former to understand the patterns of international cooperation between innovation clusters [6], ii) the latter was to analyse how to create bridges for cooperation between these clusters [7]. In this way, it was possible to orientate towards the pursuit of objectives through project management methodology and the best way to measure these collaborations in an R&D project environment. Finally, the definition of the objectives to be monitored for the success of an R&D project through Project Management Metrics and a data-driven analysis with three different approaches, Project Success, EQM and Key Performance Indicators (KPIs), were studied. The analysis of Project Success was carried out from two points of view, the first one ensuring the success of an R&D project through the study of the project management life cycle [8]. Here, a hybrid project management methodology was proposed through the analysis of the success dimensions of one R&D organisation, analysis and interpretation of information and scientific literature available at the time; the results were published in a paper presented at a scientific conference [9]. The second component of project success was achieved by conducting a study using Q-Methodology and semi-structured interviews to define the success criteria of the projects carried out in one R&D organisation from the point of view of the organisation and its stakeholders [10]. For the study of KPIs, research was carried out on identifying these indicators for project-based organisations through a lean approach, and the results of its development and implementation were published in a paper included in a scientific journal [11]. Finally, for the study of EQM, its analysis and the study of its use in R&D projects was included in a journal paper summarising the research methodology created. 4 The methodology was implemented and tested in three different projects (An expanded description of the projects can be found in Section 4.1.8.4). The first project with internal funding (UPV/EHU) in which a real-time machine monitoring platform was developed. The results have been presented in a research paper at an academic conference [12]. A second project with regional funding (Elkartek5), in which the aim is to predict the wear of cutting tools in broaching operations. The results are being 3https://www.ehu.eus/en/web/cfaa/home - Last Access: 20/12/2022 4Paper in the process of publication 5https://www.spri.eus/en/ekogarapena/ - Last Access: 20/12/2022 1.2. MOTIVATION. 9 collected, and the development and conclusions will be presented in a research paper under preparation. Furthermore, the third project was developing a European-funded project task (InterQ Project - Horizon 20206). The job was to create virtual sensors for the manufacturing process of a machine through the analysis of available quality and process data. In addition, one journal paper was published with the development and conclusions of the machine assembly for data collection, and analysis [13]. In addition, we are developing a second journal paper resuming the descriptions of the virtual sensors, data analysis, and conclusions of the research. We present the papers published at academic conferences and in scientific journals in this thesis, accompanied by the papers presented. 1.2.1 Personal motivation. The motivation for researching this topic stems from the awareness of the need to use better the resources allocated to research. The research’s main objective was to find a way to improve the efficiency of the resources available during research. Contrary to what experience dictated, the researcher set out to find ways in which R&D projects are more researcher-friendly, valuable, high-quality, classifiable and defensible results are obtained, and the money invested bears fruit that can benefit society. There are many ways to do this. First, data analysis alone can help the administrations and control bodies of companies and research centres to use resources allocated to research and development. However, the denaturalisation that comes with pragmatic data analysis dehumanises project management. Above all, project management is based on managing people rather than developing a data pipeline to get a job done. Therefore, a mix of the two worlds, data-driven and project management, is an approach that can be beneficial to the discipline and can help to fulfil the motivation. 6https://interq-project.eu/ - Last Access: 20/12/2022 2 Theoretical framework and methodology. 2.1 Introduction. Based on the concepts developed in the previous Section (See Section 1.1), this theoretical framework aims to set up the basis to formulate a new data-applied and knowledgebased project management methodology for R&D projects on KTTOs, which demands an understanding of: •University-Industry collaboration (UIC) characteristics; •TRL function between technology and knowledge transfer; •Project management metrics for R&D projects: – Project Success; – KPIs; – quality measurements. •Project management methodologies for R&D projects. 12 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. 2.2 Theoretical Framework. 2.2.1 University-Industry collaboration. Dramatic changes in the last decades have transformed the organisation of modern life, and universities, as a fundamental part of modern society, have not been immune to those changes. Universities serve a critical role in training the next generation of specialised, informed individuals while disseminating information. On the other hand, companies are swiftly adapting to changing situations, analysing and evaluating the risks and possibilities they face. 2.2.1.1 Contextual perspective. During the 1980s, Europe feared losing its leadership role to emerging countries due to economic slumps. The European Commission thrives on regenerating the technology program to create competitive European industries with the active collaboration of universities to increase investment in applied research, development, and innovation activities. Governments at all levels urged universities to make more outstanding contributions to their national innovation systems. As a result, universities flourished as essential players in regional development activities, which eased innovation-based growth [27]. New roles have emerged for universities alongside the growing importance of knowledge production and innovation, creating both the opportunity for and the necessity to rethink the meaning of Universities as the active member of the society they were. Privileges, roles, resources, tasks, and duties needed to be analysed and reformed to occupy and fulfil the role they were given. Even though that UIC is far from being a novelty [28], over the past decades, efforts to enhance this collaboration have been widely supported. Local companies, governments, and society pressure universities to be more involved and relevant. Universities responded accordingly by opening themselves up to external agencies and actors, engaging with society and increasing their contributions, facilitating the emergence of a ‘Third mission’ [29]. However, universities were not only asked for involvement and relevance but were also charged with tasks involving legitimacy, governance, marketisation, internationalisation and exploitation of higher education results [30]. Internationalisation for universities arrived in the Bologna Agreement, which located them into structural reforms of their programs and curricula to enhance consensus and alikeness among degrees across Europe, giving students far more options and more diversity in planning programs. Furthermore, the Bologna Agreement increased access to research collaborators and opened universities up to disputes with and comparisons against universities in other countries [14]. Commercialising academic knowledge is one of the best ways to generate the academic impact pursued by universities due to the easy measurement of the market acceptance for the outputs of academic research [31]. Similarly, universities are setting up Technology 2.2. THEORETICAL FRAMEWORK. 13 Transfer Offices (TTOs) with internal support functions and procedures to foster the technology transfer process to industries. The third mission is referred to Universities’ social, entrepreneurial, and innovative activities, in addition to their teaching and research missions that result in additional societal advantages [32]. The previous years have seen an increasing emphasis on improving activities related to this mission, contributing to changing their stakeholder expectations of what universities can achieve. Besides the traditional two pillars in academic teaching and scientific research, the Third mission is about how consciously and strategically universities contribute and deliver benefits for their societies in four significant areas of activity: continuous education, technology transfer, innovation, and social engagement [29]. However, despite all this, the idea of the third mission emerged from within the system. Moreover, it emerged as a university response to a broader set of drivers motivated by a different set of aspects, such as an increasing need for funding, scientific knowledge impact, knowledge production and competitiveness Figure 2.1. Fig. 2.1: Drivers for university third mission [14]. Hurmelinna [33] has listed a wide variety of potential motivations for UIC, which vary according to the company’s size, culture or geographical location. At the university side, they can be summarised (but not restricted) to the enhancement of teaching, access 14 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. to funding/financial resources, the source of knowledge and empirical data, political pressure, reputation improvement, and job offers for graduates [34]. For universities, not engaging in such collaborations takes them away from these benefits and creates an atmosphere of isolation which, due to an increasingly globalised environment, cannot be afforded. 2.2.1.2 About innovation management. According to Porter [35], territories need to develop innovative strategies to build competitive advantages based on their existing resources, skills, capacities, and trends. The relationship between the university, government and industry (triple helix model [36]) is considered an essential element for the third mission. Furthermore, it is beneficial for classical universities because the realisation of transdisciplinary activities conducted outside the university helps them to create applied research and education, making them more realistic, applicable, and relevant for society and economy [37]. Traditionally, industries enunciate a problem situation to universities and wait for a solution because of the research carried out. However, this has not been satisfactory and lacks the involvement of critical stakeholders and a clear mission, order, and vision from both sides. Nowadays, the company usually sees the importance of this collaboration to obtain more successful and innovative research results [34]. As UIC develops, society and industries are increasingly motivated by the added value of their products and processes and the opportunities arising from the intense collaboration. This has caused a dramatic change for universities and transformed them into open institutions that became active players within the regional innovation system. The necessity of enhancing knowledge transfer between public research institutes and business was recognised by the European Commission (EC) as one of the ten crucial areas of action in the European Union’s (EU) innovation policy [38]. In terms of product-supplier relations and access to tacit and explicit knowledge and labour supply, universities and local governments enhance local companies’ innovation processes and become a sustainable source of practical knowledge and a driving force of technology exchange [6]. Although companies now recognise knowledge as a fundamental asset and the primary resource to boost innovation and increase productivity through knowledge exchange, the acquisition and absorption of external knowledge, resources, and technology are challenging because their producers and users come from different environments [39]. This knowledge allows companies to raise several diversification strategies supported by local governments pursuing their transformation based on competitiveness through efficiency to one based on innovation. According to UNE 166001:2006, innovation is the ”application of new or significantly improved methods, techniques, or supplies in any activity whose objective is to obtain new products or processes or significant improvements in existing ones” [40]. Despite 2.2. THEORETICAL FRAMEWORK. 15 this clear definition, in the academic literature, there is no joint agreement about why, where, and how it occurs [41]. Innovation per se is not an isolated process. It is composed of the interaction between stakeholders with different knowledge, experience, and understanding, and depends on a complex mixture of factors [42] like: •Economic circumstances; •company maturity level; •government support; •university capacity for R&D processes; •access to qualified personnel, among others. However, recent innovation processes have become more difficult due to pressure from the industry for faster technological evolution to compete with international markets and the shortening of the product life cycle. In comparison, the university must grow in technical knowledge and fulfil a broader social role. Likewise, research clusters focused on technology transfer are required to increase funding for new research activities and equipment to generate theoretical and practical knowledge. 2.2.1.3 University-Industry Collaboration’ Characteristics. Policymakers tend to focus on short-period investments where they can achieve results as soon as possible as a booster to their campaigns in their next elections. Naturally, more attention is being paid to organisations that act as KTTOs intermediaries in the innovation process that could accelerate the obtaining of results. The efficiency of these collaborations is a crucial issue for policymakers [27] because the results achieved can be easily transferred to the industry quickly and effectively, contributing to the growth and good health of the economy. This reflects on universities by forcing them to develop collaboration strategies with regional industries and other universities and with the industries or university members of the international innovation ecosystem. Publicly or privately (or a combination of the two) funded KTTOs housed in universities or public research organisations; publicly funded regional economic development agencies; knowledge-intensive business services organisations; professional associations; advisory bodies; or knowledge workers; could all be viewed as intermediaries that facilitate knowledge transfer in support of the innovation process in businesses [43]. These intermediaries are ”organisations or bodies that act as agents of brokers in any aspect of the innovation process between two or more parties” and are crucial nodes connecting suppliers to the users of knowledge [44]. All types of innovation intermediaries share four functions [45, 46, 47]: 1. to connect actors between universities, industries, and governments; 22 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. Level 5 component and breadboard validated in the simulated or real-space environment. Level 6 System adequacy validated in a simulated environment. Level 7 System adequacy validated in space. Table 2.1: Original TRL model [16] As has been described, TRL takes a particular technology from the approach of fundamental principles, validation of the concept, demonstration through a prototype, and successful operation. The main goal of the tool is to include external skills that can provide new information and experience essential for the growth and improvement of the innovative capacities of businesses and institutions. However, TRL is used to understand the maturity of the technology and to communicate the maturity of a methodology, concept, and development of internal and external projects. Companies have been using TRL since the mid-1990s and have been adapting the tool to obtain government funding for R&D projects. Some authors have described that TRL classification can serve not only as a measure of the maturity of a technology but also as a measure of its readiness to be integrated into a more extensive system. TRL is currently being used to monitor the maturation process within a technology development process and to state expectations on categorizing research projects. Moreover, knowing the status of the research serves as input for establishing long-term partnerships and research initiatives within the company. There is a great deal of empirical research indicating the importance of external sources of knowledge, resources, and technology in the development of innovation in companies [70], and not just in NASA. Even though much of a country’s capability for innovation resides at universities, the tool is not commonly used in academia and is only used in some collaborative research projects. Moreover, the tool is expanding, and those who have used it describe it as a helpful instrument for communicating with the academy regarding the development of processes and their different stages. As a result, its use has become more popular, and those companies who have used it describe it as helpful in understanding the university regarding the development processes and the different stages of the research project [42]. Depending on the sector or the objective in which the TRL is applied, it is optional to reach all levels. The technology, system or process may be valuable if it only reaches the basic levels of the TRL. However, the results are only valuable if they reach the final stages of the tool. Each stage of the TRL has associated risk criteria depending on the level at which it is being worked. The risks of putting a successful prototype in actual conditions differ from those of previously submitting the same prototype for evaluation into simulation software. Understanding, describing, and controlling the evolution of these risks, depending on the case of the technology or the project to be completed, is a critical task for correct management through the project life cycle and to ensure that it achieves the desired benefits. However, when considering these risks, not only those 2.2. THEORETICAL FRAMEWORK. 23 associated with the development of the technology (inner picture) must be considered, but also the risks associated with the adoption of the technology (broader picture), which in many cases may determine the use or not of the final result of the project. In the 1990s, the TRL evolved to a nine-level classification to broaden the vision and be clearer at each level. At that time, the TRL was clearer as a tool to monitor and evaluate the technology development process, to provide a criterion for categorizing research projects and a scale to compare technologies. Furthermore, it provides stakeholders with a single language to comprehend the underlying technology, the technological demonstration to be achieved, the product prototype or product project development, and, most importantly, to assess the efficacy of technology transfer. Although it was initially intended only for NASA suppliers [68], long-term relationships could be built for developing a specific technology or research project. However, despite its spatial focus, its uses soon became common because its final results could be easily adapted to assess the readiness of a particular technology, product or system required for and specific objective in almost any sector. Each level within the classification is essential, as they lay the foundation for progress at subsequent levels and provide information to make future decisions. The tool is based on engineering assessments to communicate signs of progress and hypotheses and categorize them within the process management and external stakeholders’ point of view. 2.2.2.1 Definition of Technology Readiness Levels. A general outline of each level will be described to understand the relationship between the stages. Figure 2.2 provides an overview of the TRL and the phases of development. TRL 1: Basic principle observed and reported. The bottom layer of TRL is where the basis of the evolution or the use of a particular technology is set. Basic scientific research starts at this level by observing and reporting some or any exciting characteristics about a particular subject (material, programming language, the usability of technology, and others.). For example, Additive Manufacturing (AM) technology has acquired relevance and interest in academia and industry during previous years because it allows the creation of complex geometries with customizable material properties. A practical example of the TRL 1 related to AM would be a KTTO, university, or company noticing the design versatility of the technology, the potential lightness of the structure, or the material properties produced by the technology. TRL 2: Technology concepts and application formulated. Once the basic principles have been observed, the process can be formulated to identify some potential uses of the topic in question (material, programming language, the usability of technology, and others.). At this level, the uses or applications are still entirely speculative, and supporting the formulations with more specific experiments or detailed analyses is still optional. 24 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. Fig. 2.2: TRL and phases of development, based on [15]. Continuing with our example of AM technology, the next step that fits at this level is replacing an active component of a complex system, such as an engine component, with a part designed and manufactured using AM. TRL 3: Analytical and experimental critical functions and characteristic proof-of-concept. At this level, once the conceived concept has been formulated, an active research and development process is initiated to bring the idea to maturity. It is then necessary to include analytical studies to place the technology in an appropriate context and to carry out laboratory verification to validate that the analytical predictions are physically correct. In a few words, what is sought on this level is the proof-of-concept’ validation through analytical and experimental approaches to applications or concepts formulated at the previous level. For our example of AM technology, the next step that fits this level is the design of the piece or component, validated through CAD/CAM/CAE software simulation. TRL 4: Component and breadboard validation in a laboratory environment. Once the ’proof-of-concept’ is analytically and experimentally validated, the process of confirming its functionality inside the system starts. This process must serve as the best support of the concept formulated and should accomplish the requirements of the potential system applications. To carry out this activity, specialized organizations or companies should participate side-by-side in the evolution of the concept. Not just because of the knowledge and experience gained by participating throughout the evolution of the concept but also because, at this level, the cost of the research starts to rise (depending on the technology), so some formal sponsorship should be sought and attained, for example, through governments or industry investments. 2.2. THEORETICAL FRAMEWORK. 25 In our example of AM technology, the study of the piece or concept’s manufacturing parameters should be analyzed and completed. At the same time, validation within the final ensemble should be carried out to ensure that the characteristics of the concept meet the requirements of the final system in which it will be installed. TRL 5: Component and breadboard validation in a relevant environment. The accuracy of the tested concept is significantly improved by experimentation at TRL 4, enabling integration into the system with suitable supporting components and enabling testing of the full implementation of the entire system in a simulated or realistic setting. This process may include one to several new technologies in the demonstration. This activity should be carried out in specialized facilities only available to formal R&D organizations or corporate laboratories. However, it should also involve formal sponsorship needed in the previous TRL. Communication between the various project teams, both within the organization in charge of the development of the project and the organization that will receive the development, needs to be fluid and understood at different levels. Many of the failures in technology transfer start at this level, so a project management approach must be implemented to help solve the problems that may arise in using the prototype or understanding the development and testing process for the final application. This demands high technology expertise (development and implementation) of the people involved in this process should be high and commensurate with the project’s needs. Following the design of our concept with AM technologies, the next step that fits this level is the part’s manufacturing. AM printers are expensive devices that people with specific knowledge can only operate. Furthermore, the necessary facilities for powder handling, cleaning, and measuring the part are not easy to obtain due to their high cost, so this process must be done in an R&D centre or KTTO with this capability or in a company’s AM lab. TRL 6: System/sub-system model or prototype demonstration in a relevant environment. The result of the TRL 5 is a piece, a proven concept, a representative model, or a prototype ready to be tested for a technology demonstration in a relevant environment. This environment depends entirely on the developed concepts and is also aligned with the project’s cost. The demonstration may represent a natural system application, or it may merely be a close representation of the intended application while using the same technology. This task demands new technologies to be involved in the demonstration. Due to that reason, this activity should be carried out on specialized facilities only available to formal R&D organizations, KTTOs or corporate laboratories. However, it should also involve formal sponsorship needed in the previous TRL because of the increasing costs. For example, according to Mangkins [71], this activity should be carried out by appropriate formal projectized organizations that can successfully manage the objectives within the time, cost and scope required. 26 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. For our concept developed with AM technology, which at this level is a satisfactorily manufactured component that fulfils the manufacturing requirements, it is now the time for it to be further tested. At this level, the piece’s porosity, traction, roughness, fatigue, and hardness analysis should be completed to prove that it is in line with the requirements to be installed on the final assembly. TRL 7: System prototype demonstration in the expected operational environment. The main objective of TRL 7 is that the system, component, prototype, or model completes a demonstration in the expected operational environment. At this level, the component should be near or at the scale of the planned operational system, and the demonstration must occur in the actual expected operational environment. This is done to assure the system engineering and development management confidence, one step further than the purpose of technology R&D. Once again, this activity can only be carried out in specialized facilities only available to formal R&D organizations, KTTOs or corporate laboratories, but should also involve some of the formal sponsorship needed in the previous TRL. Moreover, it should be carried out by appropriate formal projectized organizations that can successfully manage the objectives within the time, scope, quality, and cost requirements. In our example of the tested prototype, the activities that should be carried out at this level to meet the objective are the analysis of the components under simulated operating conditions, such as temperature changes, currents through the part, and vibrations experienced, among others. TRL 8: Actual system was completed and ‘qualified’ through tests and demonstration. Once the technology reaches this level, the system development of most technology items is done. Once the prototype has been demonstrated to meet the system’s criteria, decisions may be made to incorporate it into an existing system or to construct a completely new system based on the prototype. Naturally, these specialized tasks and decisions can only be made at appropriate specialized facilities available to formal R&D organizations, KTTOs or corporate laboratories. However, they should involve some formal sponsorship needed in the previous TRL. Furthermore, it should be carried out by appropriate formal projectized organizations that can successfully manage the objectives within the time, scope, and cost requirements. For our example of the component manufactured by AM Technologies, at this level, our prototype has passed the various safety and performance tests carried out at specialized sites close to or like the last factory where it would be produced. TRL 9: Actual system flight proved’ through successful mission operations. All technologies successfully being used in existing systems in any industry have passed the TRL 9. At this level, once the prototype has passed all tests and proved that it can be safely integrated with the system and will fulfil the performance requirements, it is installed and tested under operational conditions. This operation can generate new sets of 2.2. THEORETICAL FRAMEWORK. 27 bug-fixing processes not identified previously by the team in charge of the development. However, this bug-fixing should be manageable if all previous TRLs were passed. Finally, in our example, the component designed and manufactured utilizing AM technologies can be safely installed on the final planned system, and the bug-fixing process should start. Despite the success of the TRL classification, some authors state some issues regarding the use of the tool. Sauser et al. [72] stated that the tool ”does not include any guidance for the uncertainty that may be expected in moving through the maturation of TRL or that it does not compare with any other alternative to TRL” [73, 74]. Further descriptions were needed to clarify the scope of each level due to the original description of the tool was considered vague, causing ambiguity in the understanding between the researchers, management and potential program users. The central and somewhat restrictive aspect that has led to significant developments and expansions in the use and description of TRL is that it is concerned only with assessing the maturity of individual technology, leaving aside where that technology is located within a more extensive system or how it integrates with other technologies. The TRL scale was enhanced further in 1995 with the articulation of the first definitions of each level, coupled with examples to help with comprehension [75]. Despite its relative success, the Department of Defense of the United States introduced the concept of Manufacturing Readiness Level (MRL) to streamline technology transfer in more dynamic manufacturing and to expand the original TRL to incorporate concerns about production and risks associated with time and manufacture of parts, components, systems, and technologies. The MRL is a metric that ensures that the development of engineering, design process and the maturation of technology can be associated with a manufacturing process facilitating a quick and easy transition and communication with the stakeholders involved in the project. 2.2.2.2 About project management for Technology Readiness Levels 5 to 7. Developing new technologies often depends on the previous success of advanced technology research and development efforts. The correct use of the TRL classification can also be helpful not just to know which skills have been acquired but also to know if these skills can be used not only in a specific project but also in different projects. These developments inevitably lead to four significant challenges in each project, performance, quality, schedule, and budget. First, with a correct risk analysis, advanced technology development projects reduce the uncertainty in the so-called ’Iron Triangle’. With such measures, project progress may be improved by cost overruns, schedule shortfalls, and the gradual erosion of initial performance goals [75]. The challenge for PMs is to determine technology readiness and risk assessment in a clear and well-documented way and to be able to do so within the precise stages of the project life cycle. 28 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. As mentioned by Mankins [71] for TRL 5 onwards, a projectized approach is necessary to manage the project’s development effectively. At this point, a previous estimation of the objectives pursued in terms of cost, time or scope/quality can be stated. The majority of project development risks start during these stages because the project has now entered an experimental phase and has left the research phase behind. Time and cost control within R&D projects has been a constant concern for companies, governments, universities, KTTOS, research clusters, and researchers. The pace with which a concept develops and the expenses associated with project development is easier to regulate in contexts where time, objectives, and costs can be specified nearly from the beginning. However, in R&D, even when the time or cost constraints are exceeded, the project can be considered a failure. To increase the uncertainty in R&D projects, even if time, cost or scope are fulfilled, only sometimes the quality of the research is adequate for the result of the project, or the different activities carried out during its life cycle. Keeping track of all the stages and activities carried out in the project is one way in which the quality of the project’s outcome can be, and again, only in some cases, considered a success. However, a further problem arises from this, measures of success of R&D projects are associated with many internal and external factors that go beyond the iron triangle, as explained above. On the other hand, the organizational strategy of universities and KTTOs in terms of developing research lines, participation in projects or searching for alliances with companies are decisive points for their continuity. Therefore, the definition of KPIs for an organization that carries out high TRL projects is vital for both the projects’ success and the collaborations’ success. For these reasons, a common understanding of project control metrics, fluid communication, as well as defined criteria that determine the success of a project and quality assessment throughout the project, and the KPIs of the organizations carrying out these projects (on the understanding that companies have previously defined and actively monitor their own KPIs) is vital in order to delineate a suitable project management methodology for R&D projects and the success of the technology transfer process on this TRLs. 2.2.3 Project management metrics and Key Performance Indicators for R&D projects. The EU’s attempts to accomplish the digital transformation of the economy have gained strength to regain the privileged position in the world that it held years ago. Industry 4.0, the solid technological bet made by Eastern countries in collaboration with China, and the increasing pace of innovation in economies such as the United States, combined with the economic crisis caused by the Covid-19 pandemic, will leave in the coming years the most significant investment in R&D in modern history. 1. Funds that will reach local 1EUR 24 billion worth of non-repayable grants from the EU: Poland´s most extensive cohesion policy programme approved by the European Commission - shorturl.at/tACNR 2.2. THEORETICAL FRAMEWORK. 29 governments to support the transformation of their economy by supporting technological projects led by industry in collaboration with universities. Due to the current environment and the amount of funding available for their implementation, it is essential to regularly monitor how these resources are used for the projects’ development and the outcomes produced by UIC. However, since the beginning of the 1990s (with the first study carried out to measure the performance of projects carried out in a UIC [76]), few other attempts have been made in this area despite its importance for the entities involved and policymakers. Furthermore, digital transformation and development of highly technical projects are characterised by being part of a highly interconnected set of subsystems with high costs, produced at a low volume, that require comprehensive and profound knowledge and skills, involving multiple collaborators and maintaining a continuous integration between the customer and the supplier. Nowadays, a fully structured and widely accepted system of indicators is still needed to evaluate the results of the UIC [77]. A first attempt to measure the performance of collaborations for implementing R&D programmes and projects, together with a method for measuring it, was made by Fernandes et al. [26]. On the other hand, the study is based on the creation and development of a theoretical technique but needs an actual demonstration or validation. Perkmann et al. [78] identified four stages of UIC, developed a success map explaining how these collaborations work and identified cause-and-effect relationships for their success. In addition, a set of performance indicators was proposed for each collaboration stage. Current project management tools and techniques have proven inadequate and insufficient for monitoring the development of highly technical and R&D projects. If these are not faithfully controlled and measured during the project life cycle, they can act against their standard development [79]. Additionally, innovation must be evaluated using a wide range of indicators because it is a multidimensional and complex notion that does not fit with typical measurements [80]. Additionally, the complexity of innovation increases with the heterogeneity of the parties involved in a UIC, as we have already described in the contextual perspective of this type of collaboration (2.2.1.1). Some research suggests that measurements can be beneficial for innovation, arguing that these measures can help managers to control tasks, processes and outcomes, ensuring that innovation is well-supported and carried out efficiently. For example, Browning & Ramasesh [81] concluded in their research that many existing models focus more on the activities performed than on interactions or project deliverables because humans tend to pay more attention to activities that can be measured, and that can lead to a concrete result. Another research line suggests that measurement can dissuade managers from seeking to deepen innovation and obtain more innovative results as a short-term reward. According to several studies, innovation measurement hinders innovation by pushing organisation members to concentrate their attention too narrowly and lose sight of the broader focus that it should have. 30 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. What should we do? Which way to go? Experience dictates, regardless of the environment we are talking about, that what cannot be measured cannot be improved. However, there is a balance to be struck. Too much measurement during the project life cycle can be detrimental to the normal development of the project, especially when these projects are developed in SMEs or small project teams or are managed by PMs who do not have the appropriate or necessary experience to manage and perform these measurements, as the excessive paperwork around the work can be overwhelming for both the PMs and the project team, which is why, for example, traditional project management methodologies cannot be applied to all types of projects, and neither do all project teams. In general, process or performance metrics to measure the maturity of new technologies and systems have yet to be fully developed. Currently, the techniques and tools available, such as Quality Function Deployment [82], Concept Selection Process [83], First Requirements Elucidator Demonstration (FRED) [84], Integrated Design Model [85], Subsystem Tradeoff Functional Equation [86], Design for Manufacturability [87], Design-Build-Test Cycle [88]. Periodic Prototyping [89], cost as an Independent Variable or CAIV [90], and Lean Product Development Flow [91], are fragmented and not used consistently throughout the process [92]. It is difficult to establish control mechanisms that can effectively affect the development of highly technical projects and R&D projects due to the lack of process metrics, high unpredictability, reliance, and lack of consistency [93]. However, can the management of R&D projects be measured? Understanding the limitations and unique characteristics of this type of project and the development of this thesis project, we believe it is possible. Some companies believe the same and measure some aspects of the process. On the other hand, researchers have also discussed the same question and have contributed to the scientific literature by debating what kind of measurement is beneficial. Still, after decades of research, the conclusions are mixed and need clarification. Having processes for planning and monitoring projects is necessary to guarantee the success of the efforts made on R&D. Cooper & Kleinschmidt [94] concluded in their research that project plans, task scheduling, monitoring, and feedback are among the ten critical factors for the successful development of a project. Dvir & Lechler [95] concluded that efficiency (schedule, money, and scope), perceived value, and customer satisfaction are all positively impacted by the planning quality. Pinto & Mantel [96] found that for R&D projects, inefficient scheduling of tasks is strongly related to failures in the implementation processes and that both monitoring and feedback can impact customer satisfaction. Furthermore, the lack of an appropriate plan makes it difficult to control the development processes that can lead to cost overruns, delays, or failures in project implementation as has already happened in some government projects, R&D efforts [97] and new product development [98]. Maintaining a comprehensive picture is especially crucial during the early phases of developing an R&D project when uncertainty is still high, but corrective measures may still be addressed. PMs must apply this approach to the project to enable them to 2.2. THEORETICAL FRAMEWORK. 31 measure the process and control the development of the system through proper planning, scheduling, and monitoring [79]. In this research, we will focus on three aspects to understand this holistic view of managing and controlling an R&D project, the study of KPIs, Project Success and Earned Quality Method (EQM). 2.2.3.1 Key Performance Indicators. At the organisational level, the development of measurement systems is necessary to set objectives and monitor the effectiveness and efficiency of the use of resources. Commonly, these metrics take the form of KPIs, which provide an objective criterion for forecasting, measuring and planning the activities carried out in the company. It should be emphasised, though, that performance metrics’ goals, definitions, and contents differ. Since they must fit the competitive environment and strategy, many approaches are utilised to design and choose business KPIs [11]. Within the scientific literature, it is possible to find several descriptions of KPIs categories. Cortes et al. [99], identified five strategic categories for KPIs: •Cost; •quality; •flexibility; •stock; •lead time. With these categories, there is an intention to capture the organisation’s strategic objectives and enable alignment with tactical, strategic, and operational performance. For example, Toor & Ogunlana [100] found different authors who included customer and stakeholder satisfaction as project success criteria in addition to the traditional iron triangle (Time, Cost, Quality). Within projects, Kerzner [101] identifies time, cost, resources, scope, quality and activities as critical metrics for project management KPIs. For example, Toor & Ogunlana [100] enhances the project team’s capacity to control project risks and find solutions to issues that arise during the project life cycle to evaluate the project’s success. Technical efficiency of execution, management and organisational implications, staff growth, partners’ technological capabilities, and organisational performance is also considered in measuring project success. However, when using these fancy metrics, conventional metrics such as cost, schedule, quality, and security should be addressed [102]. When discussing the integration of KPIs in organisations, Toor & Ogunlana [100] highlight operational, life cycle, strategic and socio-economical aspects. The author also assures that the criteria for measuring the success of projects should be based on strategy, sustainability and security. 38 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. be built up cumulatively during project implementation. The EVM concept is applied by initially determining the expected final quality and breaking it down into quality attributes that are then associated with particular project activities, thus creating a quantitative link between the performance within the project and the resulting quality and allowing quality objectives to be monitored. It was the method of Paquin et al. [3], which introduced the concept that quality is a measurable concept that is progressively built up over the life cycle of the project, from which several methodologies have been derived ([125, 126, 127, 128, 129, 130]). However, they have yet to be directed towards R&D projects. Only Schuh et al. [129] used the principles set out by Paquin et al. [3] to assess the effects of deviations in activities on the objectives of a development project. EQM has several features that make it useful for use in R&D projects. For example, it allows PMs to elucidate and structure customer needs and expectations as it decomposes overall customer satisfaction into a hierarchical structure of quality criteria. Furthermore, according to the principle that quality is achieved progressively throughout the project, the method can aggregate lower-level quality criteria into higher-level quality objectives. On the other hand, the method provides a method for evaluating the planned and earned quality of the project deliverable throughout its life cycle. Because the quality criteria, which were jointly established by the PMs and stakeholders and against which the project activities are evaluated, are achieved gradually and cumulatively as the project is carried out, making it easier to estimate the effort required to complete them. Additionally, it offers measures of quality deviations. Finally, decisions on the trade-off between quality, time, and cost are made more accessible in this way [131]. In order to successfully carry out an R&D project, the PM must initiate an interactive, flexible and responsible control of the project plan, task scheduling, monitoring and evaluation from the beginning. The plan should be carried out in a manner that is consistent with the expected project results, should not be unduly constraining, and must be based on milestones. However, as Magnaye et al. [79] conclude, there should already be a high level of control over the process during the development of the early stages of the project, as well as a greater emphasis on the application of the chosen process metrics and lessons learned, in order to identify problem areas and address them quickly. So, thanks to the research that has been going on, the EQM approach is a suitable approach to manage this type of project because of the characteristics we have already mentioned. However, there are several aspects to be taken into account when applying EQM to a project: 1. It is necessary to guarantee that the project’s progress is shown graphically; 2. the evaluation of the quality of the tasks is subjective, so it is necessary to explore objective techniques and measures agreed upon between the PMs and the stakeholders that allow absolute clarity on the partial and final results of the project. Again, communication plays a significant role in the management of these projects; 2.2. THEORETICAL FRAMEWORK. 39 3. time and cost are not taken into account in the EQM, but knowing the project’s quality criteria in advance, and having a comprehensive understanding of it, enable PMs to explain what efforts would be required to carry out the tasks and achieve the desired quality; The need for more clarification on determining the quality criteria for assessing the project, according to the study by Paquin et al. [3], is notorious, so approaches aimed at solving this impasse should be implemented. Therefore, the approach and assessment of quality criteria from the point of view of PMs and stakeholders is of vital importance for the success of the project [10] and of the EVM. 2.2.4 R&D Project Management Methodology. As we have previously analysed, thanks to the policies implemented by some countries to promote and sustain UIC, many academic contributions have tried to explain, understand and justify these interactions in economic terms. However, only a few studies have been conducted to examine the factors that influence both the participation of companies and universities in R&D projects, the characteristics of these projects, and the management methods used in these projects. We must begin with the fact that the discipline of project management is constantly evolving in response to the projectification of societies and to the needs, uncertainties and changes that arise from year to year, which means that the validity and timeliness of the methodologies and practices developed are quickly being left behind. Now, speed and agility are characteristics demanded from project teams and managers as requirements for implementing the various technologies that make up the fourth Industrial Revolution. All this has led to continuous changes in standards, methodologies, practices and methods related to project management, to maintain and increase project success rates. However, there are two crucial aspects to take into account. Firstly, the management of the UIC differs from the management of the UIC projects; while for the first aspect, the contractual clauses for managing the results and intellectual property rights, the trust between the parties or the experience of previous collaborations, are some of the characteristics that determine these collaborations; for the second aspect, the management of R&D projects depends entirely on the organisations, their PMs, and the approaches used in the methodologies. Using classical, agile or hybrid methodologies brings different advantages and disadvantages to the project. There still needs to be a standard methodology in the literature to apply to this type of project. The term ”one size fits all” does not work in project management. However, a first effort was developed by the European Commission, which in 2016 launched the first edition of the PM2[132], a hybrid project management methodology applicable to any project, and developed taking into account the needs, culture and constraints of the EU to deliver solutions and benefits to organisations effectively managing work throughout the project life cycle. This could be classified as the first methodology created and approved by a governmental regulatory body to manage UIC projects internationally. However, some shortcomings in terms of the adaptability and lightness of the documentation or its lack of practicality when it comes to digitisation 40 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. and virtualisation have meant that its dissemination and use have not spread to the level initially intended. Naturally, this raises the question, ”And if the regulator, who is the one setting the restrictions and requirements, does not have the answer to the question of how to manage projects best to meet the needs they create, then where can we find it?” Unfortunately, there has yet to be a satisfactory response to this question. Although project management, by its nature, tries to focus on controlling and minimising the risk of a project deviating from the developed plan and not achieving its objectives, this is not easy to achieve. Project management involves two aspects: planning a new ’enterprise’ and executing its execution. For the former, according to the PMBOK [133], the main elements to control are time, cost, quality, resources, communications, risk and procurement. For the second aspect, informed decision-making and plan adjustments according to needs are some of the most relevant features. According to the PMBOK definition [133], a project management methodology is a system of practices, techniques, procedures and rules used by those working in a discipline, such as PRINCE2, Scrum or Kanban. In addition, project management methodologies, in general, are based on carrying out a series of activities in a specific order and applying knowledge, skills, tools and techniques to it, trying to make the most of the technical and human resources available to meet the project’s objectives. Adequate project management is the one that occurs when project planning allows foreseeing and correcting in time the most significant number of unforeseen events that may arise. However, in an R&D project, the research results may differ dramatically from what was anticipated at the beginning but are still beneficial to the organisation—because of this, applying traditional or exclusively agile methodologies to this kind of project is complex. The academic literature stresses the importance that an adequately constituted monitoring and control methodology for an R&D project should focus on the maturity of the technologies, the integration elements and the system as a whole. On the other hand, they agree on the need to maintain a non-linear structure and the idea that detailed planning at the beginning of the project is a challenging and unfruitful task without the right tools. Thus, phased life cycle approaches are necessary to visualise the project management process. A non-linear approach must be defined to allow more creativity, flexibility and changes in the overall ideas. To overcome the dynamic scope and deal with changes, the initial planning, the definition of the methodologies and technologies employed, and the requirement for an incremental and iterative research phase are all crucial. An R&D project management methodology should be flexible, define milestones based on research maturity or product development, be interactive, and be able to respond and adapt to changes in technology and requirements. This can be facilitated by an interactive project management methodology that promotes congruence of objectives and enhances learning by creating an information infrastructure with process performance metrics linked to the organisation’s strategy. 2.2. THEORETICAL FRAMEWORK. 41 Various methods have been discussed and developed to manage R&D projects. Phased Project Planning (PPP), a control mechanism for new product development, was introduced by NASA to ensure that projects are executed according to plan and delivered on time. However, this engineering-based approach needs to be faster and more bureaucratic [134]. Cooper then presented the stage management system approach [135], focusing on quality and requiring that each stage’s inputs and outputs be evaluated, tested, and approved before going on to the next stage to solve these restrictions. Although this strategy limits options for creativity and innovation, a hybrid project management methodology that combines agile and conventional methods is recommended. The characteristics of a suitable methodology, according to Kerzner [136], include the recommended level of detail, the use of templates, standardised planning, time management and cost control techniques, standardised reporting, flexibility for use across projects, flexibility for rapid development, user-understanding, acceptance, and usability within the organisation, the use of standardised project life cycle phases, and the assertion that it is based on guidelines. This can be achieved with agile, hybrid or classical methodologies and approaches (to a lesser extent). Agile approaches are now spreading across the discipline, showing signs of improving project success, and are increasingly being used in industries other than software development [4]. Preliminary project results are specified, initial goals are established, and project outcomes are continuously assessed and improved by utilising adaptive procedures in an agile approach. An essential aspect of agile methodologies is the distribution of responsibility among project members and the inclusion of project stakeholders in both formal and informal communications around the project [137]. On the other hand, hybrid approaches have similar effectiveness to purely agile approaches. While accomplishing the same goals in terms of budget, time, scope, or quality, analysis by Gemino et al. [67] revealed that hybrid and agile approaches considerably boost stakeholder satisfaction over traditional ways. For example, a hybrid project management methodology combines practices and methodologies from more than one project management approach, seeking to use the best practices from each approach to improving the overall results of the methodology created. Regarding the hybrid approaches and methodologies developed, several proposals have been developed in the scientific literature to improve the management of R&D projects. For example, Mikulskiene [20] developed an approach in which these projects were managed in two phases. The first phase, planning, was associated with issues such as human resources, stakeholders, partners and teams. In contrast, the second phase focused on the project’s technical developments. On the other hand, Mosbrooker [66] represented the project management life cycle in four phases, which included separate concepts for project planning, execution and completion. The author further recommended setting abstract objectives, maintaining flexible planning and focusing on constraints and the environment. Kerzner [112] represents the new product development life cycle in five phases: concept development, planning, 42 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. testing, implementation and closure; it also recommends the overlap between the phases and the division of long-term product development projects into smaller projects. The six-phase life cycle, which places a stronger emphasis on managing R&D projects, is likewise consistent with the methods mentioned above. Although four-, fiveor sixphase life cycles are defined with detailed information, their suitability is questioned as these approaches maintain a non-linear structure, whereas R&D projects cannot continually be developed with linear approaches. Gutierrez et al. [138] developed a methodology in which they combine classical project management theories and some of the best practices of Scrum. This methodology includes the phases of definition, design, development, testing and release, with an emphasis on rapid customer feedback derived from the Scrum approach in the development phase and control of aspects occurring in the release phase, as well as saving resources through redesign cycles, functionality and usability testing of deliverables during sprints in the development phases. With a similar approach, Cooper [139] represented the Triple A (Adaptive, Agile and Accelerated) approach, a hybrid structure of Stage-Gate and agile approach, which comes from the adaptation of the team to the context of the project, and the agility that occurs during iterations and spirals. On the other hand, du Preez & Louw [140] represented the Fugle innovation approach, which combines the staged approach with the agile approach. They rely on the innovation process being carried out internally but connected to the external environment and outsourcing, enabling overlapping stages and iterative loops. Sommer et al. [141] also introduced industrial Scrum, combining a staged approach and Scrum. In this approach, the organisational level applies the Stage gate approach, while the Scrum approach is used at the project level. However, when discussing the creation of hybrid methodologies, the question remains as to how the decision to combine two or more approaches is made. Fijin [142] represented a model in which decision-making on the combination of approaches is facilitated. The linear structure and degree of environmental control are the main axes for decisionmaking in this model. According to Werner [143], the status of an R&D project and its improvement potential can be identified by applying an integrated planning, management and control system in the R&D environment. In order to carry out efficient project monitoring and to examine the current project situation, several actors suggest a continuous assessment of sure pre-defined KPIs [144]. The method developed by Paquin et al. [3] directly addresses the challenges of controlling the quality achieved in a product development project. On the other hand, as we have seen, the requirements of each level change, as well as the recommended practices to carry out. It is crucial to know at what point of the TRL the project is carried out to avoid failing to meet the objectives planned in the project. Another important aspect is to understand what project success is and how it is evaluated according to the organisation carrying out the project and its stakeholders. Furthermore, it is vital to know not only how it is evaluated but also how this state of ”success” is reached. The analysis of data throughout the project, the ability of PMs to 2.2. THEORETICAL FRAMEWORK. 43 make timely, data-driven decisions, and the completion of deliverables as technology or product maturity levels develop without constraining project teams and managers with stringent processes and demanding and exhaustive documentation will help to meet project objectives and satisfy the various project stakeholders. Finally, the results of the Gemino et al.[67] study validate practitioners’ decisions to combine agile and traditional practices and suggest that hybrid approaches lead the way in approaches to project management. 44 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. 2.3 Research methodology. The analysis and understanding of each of the items mentioned above will help to situate us within the conceptual model of the research. The components and interconnections between these topics are described (Figure 2.3), with the final objective of the research: the development of a methodology for R&D project management using quality criteria as a performance indicator. Fig. 2.3: Conceptual model of the research In each of the publications presented in this thesis, the following data collection methods were used (Figure 2.4): 2.3. RESEARCH METHODOLOGY. 45 Fig. 2.4: Collecting methods As seen in the graph above, the research methodology followed in each publication varied, in the First Publication: ”Patterns for International Cooperation between Innovation Clusters. Cases of CFAA and ruhrvalley” (See Section 6.1 and Figure 2.5) the case study that outlined how the collaboration between the two clusters should proceed was developed as the starting point. A literature review was required to support general recommendations for cooperation, and it was then followed by a systematic analysis of the two clusters to identify any shared characteristics. This analysis then allowed the development of critical actions for the two entities’ cooperation. In the case of the Second Publication: ”Building Cooperation between Innovation Clusters Based on Competences Requirements. Case of CFAA and ruhrvalley.” (See Section 6.2 and Figure 2.6), a series of interviews on these competencies were conducted with three focus groups, divided according to experience, role, and technology knowledge, in which they were asked to identify the competences needed to 46 CHAPTER 2. THEORETICAL FRAMEWORK AND METHODOLOGY. Fig. 2.5: First Publication: ”Patterns for International Cooperation between Innovation Clusters. Cases of CFAA and ruhrvalley” - Research Methodology participate and manage a collaborative R&D project. Supported by a literature review, they answered the research questions and achieved the publication’s objective. Fig. 2.6: Second Publication: ”Building Cooperation between Innovation Clusters Based on Competences Requirements. Case of CFAA and ruhrvalley.” - Research Methodology 2.3. RESEARCH METHODOLOGY. 47 In the Third Publication: ”TRLs 5–7 advanced manufacturing centres, practical model to boost technology transfer in manufacturing.” (See Section 7.1), a benchmarking study on comparable centres in Europe was conducted to evaluate various aspects of scientific capacity, validation of results, research capacity, equipment suitability, technical quality of the equipment, availability of the equipment, and capacity of the same in order to comprehend the context in which a research cluster focused on high TRLs develops. With the data collected, an analysis was conducted and, based on a bibliographical review, a series of recommendations were made to the Basque Country for creating and strengthening this type of centre. For the Fourth Publication: ”Assessing the success of R&D projects and innovation projects through project management life cycle.” (See Section 8.1), in order to assess the effectiveness of projects completed in a research cluster from the perspective of PMs, a survey was carried out. Based on a literature review, the success criteria for R&D projects were translated into qualitative questions ranked on a Likert scale. The validation and reliability of the survey were confirmed through an analysis performed with Smart PLS software to validate the questions measuring the contribution of each dimension of project success to the success of the realised projects. With the input of the data analysis, it was possible to identify the most crucial success factors for the research cluster projects and produce recommendations for the project methodologies that would be created in the future. In the Fifth Publication: ”Project Success Criteria Evaluation for a ProjectBased organisation and Its Stakeholders—A Q-Methodology Approach.” (See Section 8.2 and Fig 2.7, a survey was conducted using Q-Methodology (a statistical semiquantitative technique) in which the PMs of the research cluster and the key stakeholders were asked to rank the previously identified success criteria in order of importance. This survey was based on two research questions, a literature review of the most critical success criteria for the realisation of R&D projects in the context of collaborative projects with public-private organisations and a literature review of the success criteria. Semistructured interviews were conducted as part of the survey to verify the applicability of the chosen criteria and gather data that would be crucial in selecting the order of the components. A subsequent analysis of the data allowed us to categorise the participants based on the similarities and differences in the participants’ perspectives on three factors (groups), as well as to determine which were the most crucial success criteria for this type of project for both stakeholders and PMs in the innovation cluster. In the Sixth Publication: ”Identification of key performance indicators in project-based organisations through the lean approach.” (See Section 8.3), a Systematic Literature Review (SLR) was carried out with the primary purpose of studying the relationship between project success, lean and performance indicators in a projectbased context. The current state-of-the-art was identified and examined, databases to be consulted, and keywords to be included in the search queries were defined. Subsequently, the identified documents were selected according to defined exclusion criteria. Once the documents to be studied had been defined, the publications were analysed using thematic analysis and synthesising the information collected. 54 CHAPTER 3. HYPOTHESIS AND OBJECTIVES. 5. to identify the KPIs of project-based organisations and how these KPIs can influence the development of a project. 4 Summary and discussion of the results. 4.1 Summary and discussion. In this section, we will summarise the conclusions and discuss the different contributions made during the development of the thesis, as seen in the Research Onion of the Thesis (Fig 4.1). The structure of this section follows the indications published in the Chapter XI. Thesis by published papers -Regulations Governing the Management of Doctoral Studies1 1https://www.ehu.eus/en/web/doktoregoa/doctoral-thesis/thesis-by-published-papers - Last Access: 20/12/2022 56 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. Fig. 4.1: Research Onion 4.1. SUMMARY AND DISCUSSION. 57 4.1.1 First Publication CP.1 - Conference Paper: ”Patterns for international cooperation between innovation clusters. Cases of CFAA and ruhrvalley”. The first publication (See Section 6.1) refers to the search for patterns for international cooperation between innovation clusters. This publication explores different possible means of collaboration between the CFAA and ruhrvalley (innovation cluster located in the Ruhr area, Germany 2). Furthermore, it shows how with the knowledge and expertise gained from the projects, the clusters can improve their capabilities and help their partners through the rapid and better-applied use of knowledge with a new set of skills from this type of collaboration. It is also mentioned how international joint bidding can help improve the relationship between the expertise of both parties and help connect regions and their partners in innovation ecosystems. This is consistent with what is stated in the academic literature (See Section 2.2.1) and the European Union [17] in terms of collaboration between R&D organisations. Apart from recommending a reinforcement of international scientific collaboration to increase scientific productivity and knowledge transfer, these collaborations are mentioned as one of the most important channels for disseminating and valuing knowledge. They also talk about how this type of collaboration generates better training for organisations’ human capital, a more significant generation of patents and more excellent scientific production, among other aspects. The publication also mentions the need for innovation clusters to gain competitive advantage, improve efficiency and show rapid, positive and valuable results can be achieved through such collaborative arrangements. We also refer to the many difficulties that need to be overcome, the lack of balance in terms of capacities to cope with valuable research, different institutional cultures, diversification of research activities, conflicts of interest, and risks generated from the publication of research results. However, it is also mentioned that these difficulties can be overcome with a concrete definition of objectives from the outset and with clear and understandable governance structures for the parties involved. The main contribution of this publication is summarised in a description of the common needs of innovation clusters and the approach to support activities to meet these needs. This publication is valuable for those innovation ecosystems that seek to establish relationships with similar clusters based on the Triple Helix model and that pursue the development of urban areas towards technology and innovation but with a focus on different industries. This publication is aligned with the fulfilment of the Third Main Objective and First Secondary Objectives of the Thesis. 2https://ruhrvalley.tech/en/ - Last Access: 20/12/2022 58 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. 4.1.2 Second Publication CP.2 - Conference Paper: ”Building Cooperation between Innovation Clusters Based on Competences Requirements. Case of CFAA and ruhrvalley”. In this second publication (See Section 6.2), we address the issue of how to build cooperation between innovation clusters based on competence requirements. As discussed in the first publication, an international collaboration between innovation clusters, as well as between public entities and industry, is sought and encouraged by the European Union. However, one of the requirements for such collaborations to work was based on a concrete definition of objectives and clear governmental structures. One of the components of this structure is the PM, which is why in this publication, we analyse what competencies are necessary for a PM and multi-skilled engineers to implement R&D projects successfully. To achieve it, a broad set of competencies is needed. Some of these cannot be taught in traditional classrooms alone but instead require a learning environment with opportunities to gain practical experience. The study was conducted in these two innovation clusters because these locations offer such opportunities during the realisation of R&D projects. In order to identify the sets of competencies required to collaborate and manage a research project and to find the skills that can be trained at the grassroots level in these clusters, interviews were conducted with different focus groups from the two clusters. The main contribution of this publication is to provide a list of competencies required at technical, professional and global levels for R&D PMs. These competencies include problem-solving skills, knowledge and skills in scientific and systematic analysis, social skills, and critical and creative thinking. The publication states that if there is a need to specifically describe prerequisites for working on a particular project, more interviews with project team members should be conducted, and assessment measures to evaluate required competencies should be developed. We now understand that using artificial intelligence techniques for natural language processing might be beneficial for determining people’s competencies based on the analysis of their CVs, publications, and project descriptions, among other [145]. This publication is aligned with the fulfilment of the Third Main Objective of the Thesis. With these first two publications, we sought to analyse the main characteristics of innovation clusters, we looked for patterns to create collaboration strategies, and we analysed which sets of competencies PMs should have to participate in this type of collaboration. We then moved on to the next step, which involved figuring out how to transfer the knowledge and findings from projects carried out within and among innovation clusters and the outcomes of cross-sector collaboration between the public and private sectors. 4.1. SUMMARY AND DISCUSSION. 59 4.1.3 Third Publication JP.1 - Original Journal Paper: ”TRLs 5–7 advanced manufacturing centres, practical model to boost technology transfer in manufacturing”. The third publication (See Section 7.1) is an original journal article which refers to a study carried out to create new advanced manufacturing centres whose activities focus on TRL 5 - 7. This publication sought to integrate small and medium-sized enterprises into the supply chain that can benefit from collaboration with universities and other research institutions by accessing shared and specialised knowledge. Understanding this type of innovation cluster’s characteristics, the publication looked at how to integrate more components into the aeronautical manufacturing value chain through a project management approach. The researchers applied it to the aeronautical sector in the Basque Country, identifying the necessary procedures needed for the success of these efforts and the knowledge domains that can make or break them. We show throughout the publication that advanced manufacturing research centres can be a great solution to this problem in industrial sectors with structural knowledge and specialised skills deficits. We focus on the aerospace, machine tool and other supply chain sectors, as these sectors invest heavily in R&D. However, we were able to determine that most SMEs in the Basque Country need more fixed R&D structures to develop their research activities. Despite the significant efforts made by the public administration to improve universities and training centres and raise student qualifications, they collaborate with local companies to identify the significant failures of recent graduates. One of the main conclusions of this study is to recommend that the ultimate goal of these research centres focused on advanced manufacturing should not be concentrated on horizontal developments of specific manufacturing technologies but should seek to engage in a wide range of manufacturing processes; integrating the development of process activities such as tool modelling, simulation, adaptive control of operation flow, automation, among others, reaching a level of maturity that allows the rapid and optimal flow of technology, avoiding risks in the technology portfolio of the partner companies. This way, industrial partners can undertake applied technological activities with a high probability of success. The support of local, regional, national, and European governmental bodies is a crucial component of this ecosystem because sharing public and private funds can raise profit margins and partners’ ability for R&D, which improves the likelihood of success for these kinds of projects. This article has also been used as a starting point for examining the effects of the various factors impacting project management in facilities like these (the implementation of projects between public-private entities, the correct transfer of technology, funding and the different requirements for accessing and responding to these, as well as the quality of the research, carried out). Moreover, how this strategy can enhance project outcomes. 60 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. This publication is aligned with the fulfilment of the Second Main Objective and Second Secondary Objective of the Thesis. The article mentions how the critical mass created through collaboration between companies, research centres, and universities promotes and improves the chances of obtaining funding for R&D activities, with participation in consortia in European or national projects under similar conditions with European reference centres, and with the possibility of choosing to participate in activities with a high rate of technological return, as well as access to broader funding. We also talked about how, for small companies that participate in this type of research centres, it would mean an even more significant leap forward, bearing in mind their limitations in terms of R&D investment and the difficulty of constantly developing activities for the development or improvement of technologies. We also emphasise integrating machine-tool manufacturers into the aeronautical manufacturing value chain. Participating in research projects with larger companies improves the chances of selling their products to the latter. In addition to gaining access to better financing schemes and improving proximity to the end customer, they also benefit from technological excellence inside and outside advanced manufacturing research centres. Some other conclusions that can be drawn from the study are: •Research centres around TRLs 5-7 should consist of several companies and universities and should be strongly supported by public administrations; •supply chain collaboration with Original Equipment Manufacturers (OEMs) or Tier 1 is highly recommended, with the new hub model as a vertical conception; •effort should be focused on one particular industrial sector; •research centres of advanced manufacturing can boost the relative position of a university’s research group with applied research to leading positions; •the initial list of machines and systems is key to achieving intensive use of the centre’s resources. Machines should be procured with a lifespan of at least seven years; •the location in a technology park with common services is a key aspect of the project. The common services, the environment and easy access by public transport are key; •with the idea that it should be a centre available to all partners and that management of the centre should genuinely be on behalf of the entire consortium. The centre should be managed by a university or technological agent. We have already analysed, discussed and published various contributions regarding the functioning of public-private research centres and their place within regional and European business ecosystems, as well as the various characteristics needed to achieve a more efficient transfer of research results to end companies and to define patterns of collaboration between research centres and how this helps in obtaining better scientific results, as well as boosting local economies. Additionally, we were able to emphasise the significance of project management approaches that enable the quick transfer of high- 4.1. SUMMARY AND DISCUSSION. 61 quality knowledge and the relevance of a set of PM competencies for the participation and management of R&D projects in such centres. The next step leads us to analyse these types of projects from the inside, both through the evaluation of success criteria and the assessment of this success by the research centre and its stakeholders, and the identification of KPIs for this type of organisation to improve the performance and results of the projects. 4.1.4 Fourth Publication CP.3 - Conference Paper: ”Assessing the success of R&D projects and innovation projects through project management life cycle”. In this fourth publication (See Section 8.1), we enter the world of measuring the success of projects through research related to the evaluation of the success of R&D projects and innovation projects during their life cycle. This publication is aligned with the fulfilment of the Fourth Main Objective of the thesis. In this article, we discuss how the success of R&D projects impacts technological advancement and how difficult it is to manage and govern this kind of project. Focused on improving the management of R&D projects and avoiding dramatic failures, in this research, through a data-driven approach we investigated and determined several significant criteria for measuring the success of projects during the project life cycle. The main contribution of this article is the identification of different dimensions and the evaluation, in order of importance for PMs, of success criteria for R&D projects carried out at the CFAA, as input to improve project management. These criteria were validated and evaluated by conducting a survey of the PMs of the CFAA. These results revealed that the CFAA has been relatively successful in the execution of R&D projects and innovation projects and has the potential to improve the research pipeline and platforms for new technologies, innovation and creativity in the future. Furthermore, it is emphasised that the CFAA has been relatively successful in implementing projects on time and within budget, as well as in generating new knowledge and technological products. During the evaluation of the success criteria and which were the most important for the managers, it was further concluded that since partner satisfaction was the factor that contributed least to the success of the project, customer-oriented strategies should be implemented to increase partner satisfaction, and thus improve the overall outcome of the projects. This is mainly because the internal measurement of project success can and is, in fact, different from the stakeholder’s perception of the projects. While some (the project developer) may consider a project finished on schedule and within budget to be successful, the research results may differ from what others demand (stakeholders). This is due to various factors, including the project’s timing, the ease with which the results can be 62 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. put into practice, the project’s technological urgency, or even changes in the market. One of the subjects discussed in the following article is more research into the elements influencing stakeholders’ perceptions of success. 4.1.5 Fifth Publication JP.2 - Original Journal Paper: ”Project Success Criteria Evaluation for a Project-Based Organization and Its Stakeholders - A Q-Methodology Approach”. In this fifth publication (See Section 8.2), we refer to the measurement of project success evaluated from the point of view of stakeholders and PMs, who is in charge of developing the project, through the use of Q-Methodology. In the publication, we emphasise that a project can no longer be seen solely as a temporary mission undertaken to create a single product, service or result. However, although the objective is the same, the elements surrounding the project have increased in both number and complexity. They must be viewed from a systems theory perspective [146], in which the system’s inputs and components directly determine the system’s outputs. Additionally, a system wherein the PMs are influenced by the behaviours or traditions that define an organisation may cause them to take actions that jeopardise the project’s success as measured by the stakeholders. Understanding and schematising this complexity so it can be examined in detail to increase project success rates will be one of the most significant issues facing project management in the future. One of the major concerns of organisations in terms of project management is to identify the project success criteria when evaluating or considering a project successful. With this idea in mind, the identification of a series of success criteria divided into different dimensions, defined from a literature review, and the subsequent discussion for their definition and the exercise developed using Q-Methodology gave us the possibility to create a list of these criteria for R&D projects and for that particular organisation. One of the main contributions of this publication is to identify which are the most important success criteria in collaborative public-private R&D projects, according to the point of view of the organisation’s PMs and internal stakeholders. Based on the findings, the organisation’s project managers, as well as the members of the Project Management Office, can now make plans, establish indicators and define methodologies and new checkpoints through which the success criteria identified in this study can be assessed. This evaluation can be carried out not only at the end of the project but also throughout its life cycle. [10]. Another point to note about the research conducted in this publication is the easy applicability to other types of organisations and projects. Thanks to the literature review, we have shown how important it is to know and consider the stakeholders’ point of view in determining the success of projects and how this is not exclusive to R&D organisations carrying out collaborative projects but to any project organisation. 4.1. SUMMARY AND DISCUSSION. 63 Another significant contribution of this publication is to identify the different subjective perspectives when assessing the success of projects from the point of view of stakeholders and PMs. Q-Methodology offers the possibility to classify people into different groups that share the same subjective perspectives, called ”Factors”, thanks to which three factors were defined for the organisation: •Factor 1: High quality-oriented to the output; •factor 2: Traditional Project success oriented; •factor 3: External view oriented. For example, Factor 1, comprising about 50% of the participants, is characterised by getting the job completed correctly, which includes a focus on job site safety, following official standards for project deliverables to achieve greater customer satisfaction, both in the result and in the activities undertaken to achieve it. In addition to being characterised by a desire to meet customer expectations and enhance the organisation’s reputation. In general, the objective of identifying these factors is to clarify what these points of view are and who is included in them, as well as to be able to group the participants according to their points of view. Thanks to this, it is possible to focus efforts on meeting these expectations, implement practices aimed at managing these criteria, and improve project management from different points of view. This publication is aligned with the fulfilment of the Fourth Main Objective of the Thesis. Although a Q-Methodology exercise does not require a large number of participants, one of the critical limitations of this research was the sample size because the components did not indicate a strong tendency towards any of them. As already mentioned, these results can be used to improve the approach to R&D projects in the organisation and evaluate such projects’ success. Similarly, the identified criteria can redefine new KPIs for projects and the organisation or reevaluate existing metrics already utilised, depending on the context, for projects. With the completion of these two publications, we were able to cover the topic of success measurement of R&D projects for organisations carrying out collaborative projects with public-private funding and looking for a fast technology transfer of the results. The conclusions of these publications facilitate the definition of quality criteria for projects which can be considered operational needs for the definition of Key Performance Indicators for project-based organisations. 70 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. resources available for structures such as this, as these must be directed towards the hiring of personnel, and the purchase of machines or software, processes defined in generic methodologies, such as PMBoK, are challenging to follow for small or medium-sized organisations) The definition of quality criteria can only sometimes be extended to all organisations (for some organisations, the criteria set out here may be less critical or may vary drastically). Given the complexity involved, it is a simple exercise that should be carried out in organisations where projects are carried out with the collaboration of stakeholders. We proposed a method to overcome this issue (See [10]). In addition, the categorisation of KPIs helps to guide the efforts made in the projects and take advantage of the work done to improve the organisation’s performance. The organisation can arrange for quality assurance checkpoints to be done regularly based on the number of jobs to be completed, the estimated duration of the project, or the criticality of the tasks performed. In addition, partial project reports are proposed for this control to help reduce the frequency of project meetings. The minimum acceptable quality for the work completed, as specified by the PMs and stakeholders, is one of the critical success factors for the proposed methodology and the project in general. The generation and application of corrective actions to improve quality performance or risk assurance are vital to correct the course of a project or to make decisions about the tasks performed, the personnel who perform them, the project’s objective, and the tasks programmed, among others. Task uncertainty implies low analyzability of R&D processes and makes it difficult to thoroughly plan and specify research tasks in advance [151]. As a result, there is a continuing need to collect and disseminate information to specify how to determine what is going on elsewhere and how to deal with disruptions [152]. Therefore, collaborative research projects with high task uncertainty imply that decisions must be made quickly. Consequently, a less centralised decision-making process is needed, as centralised communication patterns can cope with task-related uncertainty more effectively than hierarchical ones. Increased task uncertainty in collaborative research leads to a greater decentralisation of coordination and control practices. Apart from task uncertainty, the balanced role of R&D personnel and R&D managers is also stimulated by the technology transfer objectives of the participants [55]. The fact that this methodology has been designed and based on the information and context of the CFAA does not mean that it cannot be applied outside the organisation or to projects that are not internal. Instead, the authors believe and rely on the scientific literature to state that a comprehensive yet simple and user-friendly PM’s control of the quality of planned and executed activities is the primary input to maximise project results and achieve project objectives. Following the recommendations of Ward & Chapman [153], the generic structure of a project should be described in four phases (a project conceptualisation phase, a second 4.1. SUMMARY AND DISCUSSION. 71 planning phase, a third phase outlining the execution of tasks, and finally, a project completion phase). However, the number of stages depends on the nature of the project (and the organisation’s preparation) and might range from 4 to 8 or more [136]. According to Charvat [154], every proposed methodology should have project phases, which may vary depending on the size of the project, the organisation, or the industry, but certain typical phases should be included: concept, development, implementation, and support. According to these recommendations, the methodology proposed here is organised into four parts: definition, planning, execution and control, and evaluation and release. Based on the methodology proposed by Kerzner[150], Figure 4.2 describes the relationship between the project phases and the defined activities. 72 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. Fig. 4.2: Proposed Project Management Methodology in phases and activities 4.1. SUMMARY AND DISCUSSION. 73 The documentation generated and at what point in the project, depending on the phase, is described in Figure 4.3. 74 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. Fig. 4.3: Proposed Project management methodology including activities and documents 4.1. SUMMARY AND DISCUSSION. 75 The documentation generated during the project is of vital importance in order to be able to create different databases with project information that will serve as inputs for future projects carried out by the organisation. In addition, we propose collecting information to create the databases described in Table 11.2. We detail the phases and instruments that comprise the technique, as well as the primary objectives, actions, and outcomes for each step, following Chin’s [155] suggestions. 4.1.8.2 Phases and tools description. i Phase 1 - Project Definition. The main objective of this phase is to generate the necessary information for project planning and establish the project’s organisational infrastructure. The main objectives are: •To define the partner’s requirements for the project; •to define, together with the partner, the quality criteria on which the project will be evaluated; •to identify potential partners for the realisation of the project; •to analyse, assess and document the potential risks of project development; •to define the main objectives of the project; •to develop the first version of the project sheet; •to establish the collaboration agreement and the approval to start the project. This Phase 1 includes three activities and two decision steps: i.i Partner Requirements. The partner’s requirements for realising a project are received in this first activity. These requirements must be aligned with the organisation’s strategy and have the necessary physical and human resources to conduct the project. i.i.i External information: •Scope requirements. The scope of the project is defined, together with the main and secondary objectives of the project. •Technical requirements. The technical requirements should be defined in as much detail as possible. It will be one of the criteria for evaluating the project’s performance. •Quality Requirements. The partner should provide an initial description of the quality requirements for the project. •Time requirements. Similarly, an initial estimate of the time required to complete the project, 76 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. as required by the partner. Depending on the nature of the project, this may indicate how much time is available to complete the project. i.i.ii Internal information: •Alignment with the strategy of the organisation. Determine whether the project aligns with the organisation’s research or business strategy. •Resources available, sufficient, and qualified. The organisation should define whether it has the necessary resources to complete the project requirements. •Prioritisation of the project. The project will be prioritised depending on the project type and the organisation’s criteria. This decision can be made based on the objective and scope of the project. i.ii Quality Criteria (QC) requirements definition. The project objective, as well as the internal and external quality criteria for the project, must be defined in this activity, as well as the project managerial data (technology of the project, financing, type of project, partners involved, project code, prioritisation, Critical Success Factors (CSF) estimation; PM Assignment; Initial estimation of project hours (machines and staff, according to Partner’s requirements; technology of the project and financing). The necessary information can be converted into a checklist for the organisation for its definition and further study. This information should be included in the first version of a project sheet. The quality criteria for evaluating the project should be at most ten items. i.ii.i External information: •KPIs definition. As far as possible, the partner shall define the KPIs that will govern the project based on the technical requirements, the organisation’s success criteria and other available information. i.ii.ii Internal information: •KPI List. Based on the KPIs defined for the organisation, it is necessary to define which indicators are fed by the implementation of the project. In the same way, based on the nature of the project or the line of research to be followed, define which indicators can be fed by the implementation of the project. •Critical Success Factors. Define which Critical Success Factors affect project delivery. If available, 4.1. SUMMARY AND DISCUSSION. 77 select those CSFs defined by the organisation and jointly by the customer. [10]. •Data Analysis about previous projects. If data from previous projects are available, consult on similar projects that have been carried out, activities carried out, quality assessments, project results, and resources planned vs used, among others. An example of the databases generated using the proposed methodology is summarised in Table 11.2. i.ii.iii Documents: •Project Sheet V1. The first version of the project initiation document, with a summary of the agreements reached for the realisation of the project, will be generated. This sheet will include the project title, customers and contact details, financing, percentage of customer participation in the project, an initial estimation of machine hours and personnel, and start and estimated duration. In addition, a summary of the scope of the project, technical and quality requirements, and defined quality criteria. i.iii Requirements Definition. In this activity, the partner must be asked to approve the previously collected information and analyse the project’s feasibility jointly. An agreement between the parties must be written down in Project Sheet V1. i.iii.i External information: •Partner Feedback. The partner must approve the information contained in Project Sheet V1. If not, the partner’s requirements and the other information contained in Project Sheet V1 must be re-analysed. i.iv Strategical risks analysis and evaluation. In this activity, a strategic assessment of the project’s risks must be conducted, including a description of the risks, the severity of occurrence, an assessment of the probability of occurrence, and the proposal of initial corrective and preventive actions for the occurrence of these risks. i.iv.i Internal information: •Initial strategical risk evaluation (Description of risk; Severity and Likelihood evaluation, Corrective and Preventive Actions). An initial study of the project risks, a description of the risks encountered, an assessment of the severity and probability of occurrence, and preventive and corrective action plans for the risks encountered must be carried 78 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. out. i.v Partner Feedback. With the information from the risk analysis, the partner will request feedback to check the feasibility of the project. In disagreement, a new project risk assessment must be carried out. i.vi Risk Assessment. If the partner disagrees with the risk assessment, this activity should be performed again, taking into account the comments received and the recommendations made by the partner for approval. i.vii Project Start. If a consensus has been reached for the realisation of the project in terms of technical requirements, time, quality, personnel and other components, the project will be formally initiated. A project initiation document (Project Sheet V1) will be generated and based on this, the contractual agreements for the realisation of the project will be generated. i.vii.i Documents: •Risk Management Plan. Suppose a consensus has been reached on the risk assessment and the proposed corrective and preventive actions. In that case, this information shall be recorded in a ”Risk Management Plan” containing the definitive description of the risks, the severity and probability of occurrence, and the proposed corrective and preventive actions for their prevention. A person shall also be designated to monitor and control them. Once this has been done, the project can be formally launched. ii Phase 2 - Project Planning. This is the main phase of the proposed methodology, as it covers project planning. The main objective of Phase 2 is to delve into the description of the activities and control requirements necessary to generate correct project planning. The information collected in Phase 1 will serve as input for this phase. The main objectives are: •To carry out the description of the activities; •to define and document, together with the partner, the quality criteria management plan on which the project will be evaluated; •to plan the tasks and activities necessary to carry out the project; •to generate the necessary information for project control; •to develop the second version of the project sheet; 4.1. SUMMARY AND DISCUSSION. 79 This Phase 2 includes three activities and one decision step: ii.i Primary Planning. This activity, together with the required information on Phase 1, will result in the realisation of the WBS up to level 3. The first activities, the staff assignments and the scheduling (machine schedule) of the machines according to the prioritisation of the project will have to be planned. In addition, the project milestones must be defined according to the number and criticality of the tasks to be performed. The tasks must have information on the initial number of hours programmed, classification of the type of task (Table 11.2) and the person or group of people in charge of the task. ii.i.i Internal information: •Staff assignation. The people who will form the project team and carry out the activities will be defined. •Resource allocation. If required, the resources needed to carry out the project tasks shall be described. •Operational risks analysis and evaluation. This information will be added to the project’s Risk Management Plan. •Milestones definition. Project milestones will be defined in agreement with the partner. •Task Planning. Initial planning of project tasks and activities will be carried out. ii.i.ii Documents: •WBS Level 3. At this point, levels 1 and 2 of the WBS should have been defined. With this information, plus the information collected in planning tasks and activities, level 3 of the WBS should be defined. ii.ii QC Assessment. Based on the information defined in the activity (i.ii), no more than ten quality criteria should be defined to assess the project. These criteria should group the quality criteria defined by the partner and the organisation’s internal KPIs and Critical Success Criteria (CSC). Subsequently, and following the method described by Paquin et al. [3], the WBS -QBS will be created, which will have the information of which activities contribute to each of the defined quality criteria, as well as a valuation for that quality contribution (cj). This information should be included in the QC Management Plan, together with the periodicity of 86 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. •QC Management Plan. The QC Management Plan shall be included in the project documentation and documentation relating to changes and developments during the project. •Project evaluation. Finally, together with the project team and the PM, an evaluation of the realisation will be carried out, taking into account the achievement of the technical and quality objectives of the project, the project management, problem solving and lessons learned. iv.ii.ii Documents: •Project Closure Report. A project closure report will be created with the information collected during the project. Another advantage of using this methodology is that comparing the quality gained with the planned quality of the work performed allows PMs to detect deviations in quality and initiate corrective actions early, avoiding the unnecessary expenditure of resources. One of the main reasons why the methodology was defined in this way lies in the intention to obtain as much data as possible from the project, without this meaning more paperwork for the PM or the project team. 4.1.8.3 EQM description. The EQM described by Paquin et al. [3]. was born as a proposal to allow the PMs to assess and control the quality of the final product throughout the project life cycle. We have discussed that the EQM is based on two fundamental premises. The first is that quality is a measurable concept, and the second is that quality accumulates progressively throughout the project’s life cycle. In their method, they start by elucidating the client’s needs. To do this, the authors breaks down the overall quality objective into more detailed, lower-level objectives to clarify their meaning. This results in the Quality Breakdown Structure (QBS) that shows the hierarchy of the decomposition of the quality objectives. Partners’ preferences are then evaluated and aggregated. In the original EQM assumed no interaction between the given attributes, making the value function additive. Since the attributes and quality criteria are described in a hierarchical structure, the weights can be considered conditional evaluations. Thus the relative importance of the criterion cjto the overall quality objective wjis mathematically described by the Equation 4.1: wj= K X k=1 aksjk for j = 1, . . . , J (4.1) 4.1. SUMMARY AND DISCUSSION. 87 Where: •JThe number of criteria; •KThe number of attributes; •vkThe attribute kof the QBS; •akThe relative contribution of attribute vkto the overall quality objective ak∈[0,1] and K X k=1 ak= 1 •sjk The relative contribution of criterion cjto attribute vk,sjk ∈[0,1] and: J X j=1 Sjk = 1, for k = 1, ..., K •wjThe relative contribution of criterion cjto the overall quality objective, and: J X j=1 wj= 1. Assuming preferential independence between the tasks, the overall quality Qof the project’s final product is equal to the weighted sum of the utility value of the results xj obtained in all criteria J. Mathematically, we can write (Equation 4.2: Q= J X j=1 wjϕj(xj) (4.2) Estimating Earned Quality. Once the QBS and WBS have been determined, the PM must define the criteria to which each activity contributes. The linkage between the WBS and the QBS allows the PM to establish a relationship between the activities and the quality attributes. The PM then deepens the analysis by estimating the relative contribution of each activity to its related quality criteria. This can be done by attributing a conditional weight rij that measures the estimated relative contribution of activity aito criterion cj. The potential contribution of activity ai’s contribution qito the overall quality objective can be targeted as follows (Equation 4.3): qi= J X j=1 wjrij for i = 1, . . . , I (4.3) Where: •IThe number of activities. 88 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. •rij The estimated contribution of activity aito criterion cj, where, in addition: rij ∈[0,1] , and, I X i=1 rij = 1, for j = 1, . . . , J. Thus the contribution of each activity aito the overall quality can be decomposed into its contribution to criterion cjand the contribution of criterion cjto the overall quality. Planned Quality of Work Scheduled P QW St Planned work refers to the expected rate of completion of activities at time t, while earned work refers to the actual rate of completion of activities at time t. Planned quality refers to the expected quality that should have been accumulated at time t, while earned quality measures the actual quality accumulated at time t. The planned quality of scheduled work P QW Stmeasures the planned contribution to the overall quality target attributable to scheduled work for all activities at time t. P QW Stis defined as follows (Equation 4.4) PQWSt= I X i=1 J X j=1 wjϕj(x∗ j)r∗ ij (t) (4.4) Where: •r∗ ij (t) The expected contribution to the expected result x∗ j, as measured by criterion cj, attributable to the work scheduled for activity aiat time t, 0 ≤r∗ ij(t)≤rij They assume that the scheduled work is such that its outcome will satisfy the customer’s expectations x∗ jand, consequently, will lead to ϕjx∗ j= 1. Thus, after completion of the project P QW St= 1. Planned Quality of Work Performed P QW P t The Planned Quality of Work Performed P QW P tmeasures the planned contribution to the overall quality objective attributable to the work performed in all activities at time t.P QW Ptis defined as follows (Equation 4.5: PQWPt= I X i=1 J X j=1 wjϕjx∗ jrij (t) (4.5) Where: •rij (t) is the expected contribution to the expected result x∗ jmeasured by criterion cjattributable to the work done in activity aiat time t, 0 ≤rij(t)≤rij Consequently, the project is completed at P QW P t= 1 4.1. SUMMARY AND DISCUSSION. 89 Earned Quality of Work Performed EQW P t The Earned Quality of Work Performed EQW P tmeasures the customer’s overall satisfaction with the results obtained or the quality gained, attributable to the work performed on all activities at time t, Equation 4.6 gives the EQW P tat time t. EQWPt= I X i=1 J X j=1 wjϕj(ˆxj) ˆrij (t) (4.6) Where: •ˆxj(t) The actual result obtained with respect to the criterion cjof the work done at time t •ˆrij (t) The estimated contribution to the actual result ˆxj(t) according to criterion cj attributable to the work done in activity aiat time t. Assessment of quality deviations and initiation of corrective measures. Comparing the Earned Quality of Work Performed EQW P twith the Planned Quality of Work Performed P QW P twe obtain the quality variance (QV ) at time t(Equation 4.7: QVt= EQWPt−PQWPt(4.7) The Quality Performance Index (QPI) at time tis calculated as follows (Equation 4.8): QPIt=EQWPt PQWPt ×100 (4.8) 4.1.8.4 Practical examples of the use of the methodology. The methodology has been tested in the realisation of three projects, as mentioned at the beginning of this section. (See Section 4.1.8): iProject1: Project created for the development of a real-time machine monitoring platform. ii Project2: Project created to predict the wear of cutting tools in broaching machining. iii Project3: Project created for the development of virtual sensors for manufacturing processes through the analysis of machine data and part quality. These projects were chosen for several reasons: iSources of funding: 90 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. •For Project1, the funding was internal, i.e. it came from the CFAA funds. •For Project2, funds were used from a project submitted, approved by the Basque Government and financed by the ”Elkartek” call, an instrument within the Basque Science, Technology and Innovation Plan (PCTI 2030), which aims to support collaborative research in strategic areas of fundamental and industrial research. 4. Moreover, these are non-refundable grants. •The Project3 was developed within the framework of a European research project funded by the European Union under the EU research and innovation funding programme ”Horizon 2020” 5, which was operational from 2014 to 2020, with a budget of around 80 billion euros. ii Project governance entities: •For Project1, the main governing entity of the project was the CFAA, which was the Research Centre where the project was carried out. A report justifying the work carried out, experimental and preliminary tests at the CFAA, and documentation related to the development of the project and the infrastructure, as well as publications either in conferences or in specialised scientific journals, are therefore requested. •For Project2, the main governing entity of the project was the Basque Government, which has established minimum requirements for the acceptance of projects carried out with its funds [156]. – ”A performance report justifying the compliance with the conditions imposed in the award of the grant, indicating the activities carried out and the results obtained”. – ”Economic report justifying the cost of the activities carried out”. – ”Signed audit report drawn up by a person registered in the Official Register of Statutory Auditors under the Institute of Accounting and Auditing of Accounts.” – ”Expenditure Certification Document of the project, with the express Declaration of Concurrent Grants”. Moreover, in this project, the UPV/EHU worked with seven other regional companies that formed the research consortium. •For Project3, the main governing entity of the project was the EC, with the EC controlling and supervising the results of the projects and demanding minimum requirements for their acceptance from those who have been granted funding, among them: 4https://www.spri.eus/es/ayudas/elkartek/ - Last Access: 20/12/2022 5https://research-and-innovation.ec.europa.eu/funding/funding-opportunities/funding-programmesand-open-calls/horizon-2020en−LastAccess : 20/12/2022 4.1. SUMMARY AND DISCUSSION. 91 – Publication of research results in Open Access scientific journals and conferences. – Open access to research data. – A report justifying compliance with the technical and scientific conditions imposed in the award of the grant, indicating the activities carried out and the results obtained. – Economic report justifying the cost of the activities carried out. – Certification document of project expenditure, and others. The UPV/EHU is part of a consortium of 25 European companies and research centres in this project. The work is focused on the coordination of two of the work packages and the implementation and participation in several of the activities of the other work packages. iii Technical and scientific requirements for projects: •For Project1, the technical and scientific requirements were formulated and agreed upon by the CFAA, the Project Manager and the Principal Investigator; and finally approved by the CFAA. •For Project2, the technical and scientific requirements were formulated and agreed upon between the consortium’s eight Research Centres and Universities members, aligned with the lines of research proposed by the Basque Government for the implementation of these projects. In this case, the Basque Government approved the technical and scientific proposals of the project under its criteria. •For Project3, the technical and scientific requirements were formulated and agreed upon among the consortium’s 25 member companies and research centres. These requirements were formulated according to the guidelines of the ”Factoriesof-the-Future (FoF) Public Private Partnership” call 6, EU research and innovation funding programme ”Horizon 2020”. In this case, the European Commission finally approved the required funding aligned to the technical and scientific proposals of the project. A. Description of projects, project objectives and requirements - Activity i.i of the proposed Methodology (See section 4.1.8.2.) iProject1 - Development of a real-time machine monitoring platform. For this project, the technical and quality requirements for project performance, objectives to be achieved, and time and costs were agreed upon between the CFAA and the project’s lead researcher. 6https://www.effra.eu/factories-future - Last Access: 20/12/2022 92 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. i.i The project team consisted of four people: •Lead researcher. •Project manager. •Project supervisor. •Expert advisor. i.ii The agreed objectives of the project were to: •Create a digital twin on a CFAA machining centre. •Perform an analysis of the variables (Programmable Logic Controller (PLC) and Computer Numerical Control (CNC)) necessary to monitor the machine in question. •To achieve a scalable, high-performance implementation of the digital twin that enables near real-time data processing without loss of information. •Create a monitoring dashboard that shows the data of the digital twin and the status of the variables being processed in streaming. •Make use of 5G technologies for the virtualisation of services. i.iii The technical requirements approved for the project were: •Use open source technologies for data connectivity, processing, analysis and visualisation. •Make use of industrial protocols for data ingestion. •Development of a digital twin prototype using Spark Structured Streaming 7, running on a single node. •The program must filter and process the defined variables. •The deployed infrastructure shall be able to detect possible anomalies in the data through the detection of outliers. •Use of database for the storage of data sorted through timestamps. •Represent data through simple and intuitive graphs on a dashboard so CFAA members can understand what is happening on the machine. •Remote operation of the infrastructure hosted on virtual machines in a private 5G network. •Ensuring safe connection to the infrastructure. i.iv The quality requirements for the project were: •Detection of signal processing incidents and ensuring data persistence. •Machine signal processing with high frequency. 7What is Apache Spark Structured Streaming? - https://docs.databricks.com/structuredstreaming/index.html - Last Access: 20/12/2022 4.1. SUMMARY AND DISCUSSION. 93 •Analysis of available open-source tools. •Platform performance. •Compliance with the initial schedule. •Scientific publications generated from work done. ii Project2 - Prediction of cutting tool wear in broaching machining. For this project, the project’s technical and performance quality requirements, objectives to be achieved, and times and costs were agreed upon between the consortium formed by the Research Centres and the Universities. The Basque Government subsequently approved these. In this project, the work of the CFAA was reduced to carrying out one of the project tasks. The task was to use data analysis and the implementation of AI algorithms to predict the wear of cutting tools used in broaching machining. For this purpose, machine variables were used as features and wear as a target. Datasets from two machine tests that share similarities so that a single model is valid for both were used for the analysis. In addition, different machine learning models were examined to obtain the one that best fits the actual data. ii.i The project team consisted of six people: •Lead researcher. •Task coordinator. •Expert advisor. •Three machine technicians researchers. ii.ii The agreed objectives of the project were: •Development of a behavioural model of manufacturing processes aimed at quality prediction (surface and geometric) for broaching. •Development of functionalities for machine tool condition monitoring. •Define and develop a data architecture that integrates information related to the manufacturing process and quality characterisation. ii.iii The technical requirements approved for the project were: •Determine relationships between process and part quality variables, allowing to establish process control actions at the machine level (real-time) and factory level (early defect detection). •Creation of a monitoring system to characterise product quality. •Cutting tool wear prediction. ii.iv The quality requirements for the project were: •Analysis of machine data on edge computing devices in real-time. 94 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. •Predict cutting tool wear with high accuracy. •Compliance with the initial schedule. •Model performance. •Publications generated from work done. iii Project3 - Development of virtual sensors for manufacturing processes through analysis of machine data and part quality. For this project, the technical and quality requirements for project performance, objectives to be achieved, time and costs were agreed upon by the 25 European companies, research centres and universities forming the consortium. The European Commission subsequently approved these. For this project, the UPV/EHU was responsible for managing two work packages and participating in several tasks in six of the nine work packages. We will show the information related to completing one of these tasks. Specifically, this task consisted of developing virtual sensors (a virtual sensor is a ”pure software sensor which autonomously produces signals by combining and aggregating signals that it receives (synchronously or asynchronously) from physical, or other virtual sensors” [157]), for process control, which was able to determine the state and the different phenomena (breaks, wear, and others.) occurring in the cutting tool, based on the real-time analysis of the machine data and the data obtained from the surface quality of the part. Different artificial vision techniques were used to isolate, measure, and identify breaks and wear on the cutting tools to determine the surface quality of the part. iii.i The project team consisted of nine people: •Four researchers. •Task coordinator. •Expert advisor. •Three machine technicians researchers. iii.ii The agreed objectives of the project were: •Development of a virtual sensor for the broaching process. •Define and develop a data architecture that integrates information related to the manufacturing process and quality characterisation. iii.iii The technical requirements approved for the project were: •Determine relationships between process and tool quality variables. •To determine by creating a virtual sensor the different phenomena occurring in the cutting tool during the broaching process. •Cutting tool wear prediction. 4.1. SUMMARY AND DISCUSSION. 95 iii.iv The quality requirements for the project were: •Analysis of available machine and sensor data. •Wear data analysis. •Correlations between machine and wear data. •Compliance with the initial schedule. •Model performance. •Publications generated from work done. iv Definition of Quality Criteria Requirements for projects. Based on the information gathered from the review of KPIs (Table 11.1) and the CSC (Table 11.3) from CFAA, the quality requirements defined for each of the projects were analysed to determine which of those mentioned above were affected by the quality requirements defined for each of the projects. This information can be found in Table 4.1. ID Category Description Project 1 Project 2 Project 3 CSC - 1 CSC - Cost Management Return on Investment of the project X X X CSC - 2 CSC - Knowledge Management Knowledge generation regarding project activities (e.g., tools, techniques, approaches, processes) XXX CSC - 3 CSC - Quality Management Customer satisfaction regarding the deliverable. XXX CSC - 4 Customer satisfaction regarding the quality of delivery activities of the specific project XXX CSC - 5 Degree to which the deliverable meets its intended purpose. XXX CSC - 6 The deliverable meet the defined quality criteria. XXX CSC - 7 CSC - Risk Management Workplace Safety X X X CSC - 8 CSC - Scope Management Project goal was achieved X X X CSC - 9 CSC - Stakeholder Management Delivery activities have a good reputation X X X CSC - 10 Reputation of the organization has increased XXX KPI - 1 KPI - Financial Increase the return of investment of the projects XXX 102 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. vi Task Description definition. Following the methodology, the tasks to be carried out in the project are defined. The contributions of each of the subtasks rij of the activities aito each of the criteria cjto be assessed for the project, at level 3 and 4 of the WBS and QBS, are described: •Phase 1: ”Management and planning.” WBS Level 2 P3Ph1 - Management and Planning WBS Level 3 P3Ph1 - 1.1 P3Ph1 - 1.2 P3Ph1 - 1.3 P3Ph1 - 1.4 cj0.250 0.250 0.250 0.250 Task WBS Level 4rij P3Ph1 - 1.1.1 0.167 P3Ph1 - 1.1.2 0.167 P3Ph1 - 1.1.3 0.167 P3Ph1 - 1.1.4 0.167 0.111 P3Ph1 - 1.1.5 0.167 P3Ph1 - 1.1.6 0.167 0.111 P3Ph1 - 1.2.1 0.111 P3Ph1 - 1.2.2 0.111 0.250 P3Ph1 - 1.2.3 0.111 P3Ph1 - 1.2.4 0.111 P3Ph1 - 1.2.5 0.111 P3Ph1 - 1.2.6 0.111 P3Ph1 - 1.2.7 0.111 P3Ph1 - 1.3.1 0.250 P3Ph1 - 1.3.2 0.250 P3Ph1 - 1.3.3 0.250 P3Ph1 - 1.4.1 1.000 Table 4.6: Phase 1: QBS-WBS levels 2, 3 and 4. 4.1. SUMMARY AND DISCUSSION. 103 •Phase 2: ”Data analysis activities.” WBS Level 2 P3Ph2 - Data Analysis Activities WBS Level 3 P3Ph2 - 2.1 P3Ph2 - 2.2 cj0.500 0.500 Task WBS Level 4 rij P3Ph2 - 2.1.1 0.167 P3Ph2 - 2.1.2 0.167 P3Ph2 - 2.1.3 0.167 P3Ph2 - 2.1.4 0.167 P3Ph2 - 2.1.5 0.167 P3Ph2 - 2.1.6 0.167 P3Ph2 - 2.2.1 0.333 P3Ph2 - 2.2.2 0.333 P3Ph2 - 2.2.3 0.333 Table 4.7: Phase 2: QBS-WBS levels 2, 3 and 4. •Phase 3: ”Wear analysis activities.” WBS Level 2 P3Ph3 - Wear Analysis Activities WBS Level 3 P3Ph3 - 3.1 P3Ph3 - 3.2 cj0.500 0.500 Task WBS Level 4 rij P3Ph3 - 3.1.1 0.143 P3Ph3 - 3.1.2 0.143 P3Ph3 - 3.1.3 0.143 P3Ph3 - 3.1.4 0.143 P3Ph3 - 3.1.5 0.143 P3Ph3 - 3.1.6 0.143 P3Ph3 - 3.1.7 0.143 P3Ph3 - 3.2.1 0.200 P3Ph3 - 3.2.2 0.200 P3Ph3 - 3.2.3 0.200 P3Ph3 - 3.2.4 0.200 P3Ph3 - 3.2.5 0.200 Table 4.8: Phase 3: QBS-WBS levels 2, 3 and 4. 104 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. •Phase 4: ”Correlation and modelling.” WBS Level 2 P3Ph4 - Correlation and modelling WBS Level 3 P3Ph4 - 4.1 P3Ph4 - 4.2 cj0.500 0.500 Task WBS Level 4 rij P3Ph4 - 4.1.1 0.333 P3Ph4 - 4.1.2 0.333 P3Ph4 - 4.1.3 0.333 P3Ph4 - 4.2.1 0.500 P3Ph4 - 4.2.2 0.500 Table 4.9: Phase 4: QBS-WBS levels 2, 3 and 4. •Phase 5: ”Evaluation and release.” WBS Level 2 P3Ph5 - Evaluation and release WBS Level 3 P3Ph5 - 5.1 P3Ph5 - 5.2 P3Ph5 - 5.3 P3Ph5 - 5.4 cj0.250 0.250 0.250 0.250 Task WBS Level 4rij P3Ph5 - 5.1.1 1.000 P3Ph5 - 5.2.1 1.000 P3Ph5 - 5.3.1 0.250 P3Ph5 - 5.3.2 0.250 P3Ph5 - 5.3.3 0.250 P3Ph5 - 5.3.4 0.250 P3Ph5 - 5.4.1 0.500 P3Ph5 - 5.4.2 0.500 Table 4.10: Phase 5: QBS-WBS levels 2, 3 and 4. 4.1. SUMMARY AND DISCUSSION. 105 •Phase 6: ”Post-project evaluation.” WBS Level 2 P3Ph6 - Post project evaluation WBS Level 3 P3Ph6 - 6.1 P3Ph6 - 6.2 P3Ph6 - 6.3 P3Ph6 - 6.4 P3Ph6 - 6.5 cj0.200 0.200 0.200 0.200 0.200 Task WBS Level 4 rij P3Ph6 - 6.1.1 1.000 P3Ph6 - 6.2.1 1.000 P3Ph6 - 6.3.1 1.000 P3Ph6 - 6.4.1 1.000 P3Ph6 - 6.5.1 1.000 Table 4.11: Phase 6: QBS-WBS levels 2, 3 and 4. Gantt Chart of the project. With the information collected, we defined the Gantt chart of Project 3 and the control points (cp) for the project. Time (days) 15 30 45 60 75 90 105 120 135 150 165 180 195 210 P3Ph1 Management and planning cp cp cp cp cp cp P3Ph2 Data analysis cp cp cp cp P3Ph3 Wear analysis cp cp cp cp P3Ph4 Correlation and modelling cp cp P3Ph5 Evaluation and release cp cp cp cp P3Ph6 Post-project evaluation cp cp Table 4.12: Gantt Chart of the project 106 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. With the information collected from the control points, plus the inputs from each of the sub-tasks of the activities, the tables for the Planned Contribution of Work Scheduled (PCWS) (See section 4.1.8.3) can be created: PCWS: P3Ph1 - Management and planning. t P3QC2 (0,050) r11 = 1,0 rcp 11 (t) Weighted sum Pjwjϕjxcp jrcp 1j(t) 15 0,167 0,008 30 0,333 0,017 45 0,500 0,025 60 0,667 0,033 75 0,833 0,042 90 1,000 0,050 Table 4.13: PCWS: P3Ph1 PCWS: P3Ph2 Data analysis activities. t P3QC1 (0,150) r22 = 0,700 rcp 22 (t) P3QC6 (0,150) r23 = 0,200 rcp 23 (t) P3QC4 (0,200) r25 = 0,150 rcp 25 (t) P3QC5 (0,300) r26 = 0,200 rcp 26 (t) Weighted sum Pjwjϕjxcp jrcp 1j(t) 75 0,175 0,050 0,038 0,050 0,056 90 0,350 0,100 0,075 0,100 0,113 105 0,525 0,150 0,113 0,150 0,169 120 0,700 0,200 0,150 0,200 0,225 Table 4.14: PCWS: P3Ph2 PCWS: P3Ph3 - Wear analysis activities. t P3QC6 (0,150) r33 = 0,500 rcp 33 (t) P3QC4 (0,200) r35 = 0,150 rcp 35 (t) P3QC5 (0,300) r36 = 0,200 rcp 36 (t) Weighted sum Pjwjϕjxcp jrcp 1j(t) 105 0,125 0,038 0,050 0,041 120 0,250 0,075 0,100 0,083 135 0,375 0,113 0,150 0,124 150 0,500 0,150 0,200 0,165 Table 4.15: PCWS: P3Ph3 4.1. SUMMARY AND DISCUSSION. 107 PCWS: P3Ph4 - Correlation and modelling. t P3QC6 (0,150) r43 = 0,200 rcp 43 (t) P3QC3 (0,150) r44 = 0,500 rcp 44 (t) P3QC4 (0,200) r45 = 0,500 rcp 45 (t) P3QC5 (0,300) r46 = 0,200 rcp 46 (t) Weighted sum Pjwjϕjxcp jrcp 1j(t) 135 0,100 0,250 0,250 0,100 0,133 150 0,200 0,500 0,500 0,200 0,265 Table 4.16: PCWS: P3Ph4 PCWS: P3Ph5 - Evaluation and release. t P3QC1 (0,150) r52 = 0,300 rcp 52 (t) P3QC6 (0,150) r53 = 0,100 rcp 53 (t) P3QC3 (0,150) r54 = 0,500 rcp 54 (t) P3QC4 (0,200) r55 = 0,200 rcp 55 (t) P3QC5 (0,300) r56 = 0,200 rcp 56 (t) Weighted sum Pjwjϕjxcp jrcp 1j(t) 135 0,075 0,025 0,125 0,050 0,050 0,059 150 0,150 0,050 0,250 0,100 0,100 0,118 165 0,225 0,075 0,375 0,150 0,150 0,176 180 0,300 0,100 0,500 0,200 0,200 0,235 Table 4.17: PCWS: P3Ph5 PCWS: P3Ph6 - Post-project evaluation. t P3QC5 (0,300) r66 = 0,200 rcp 66 (t) Weighted sum Pjwjϕjxcp jrcp 1j(t) 195 0,100 0,030 210 0,200 0,060 Table 4.18: PCWS: P3Ph6 108 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. In the same way, we can define the table of the Planned Quality of the Work Scheduled (PQWS) (See Section 4.1.8.3). Activity, ai 15 30 45 60 75 90 105 120 135 150 165 180 195 210 P3Ph1 0,008 0,017 0,025 0,033 0,042 0,05 0,05 0,05 0,05 0,05 0,05 0,05 0,05 0,05 P3Ph2 - - - - 0,056 0,113 0,169 0,225 0,225 0,225 0,225 0,225 0,225 0,225 P3Ph3 - - - - - - 0,041 0,083 0,124 0,165 0,165 0,165 0,165 0,165 P3Ph4 - - - - - - - - 0,133 0,265 0,265 0,265 0,265 0,265 P3Ph5 - - - - - - - - 0,059 0,118 0,176 0,235 0,235 0,235 P3Ph6 - - - - - - - - - - - - 0,03 0,06 P QW St0,008 0,017 0,025 0,033 0,098 0,163 0,26 0,358 0,59 0,823 0,881 0,94 0,97 1 Table 4.19: Planned quality of work scheduled Now, assuming that the project starts and finishes within the planned time, it can be said that the Planned Contribution of the Work Performed (PCWP) would be: Activity, ai 15 30 45 60 75 90 105 120 135 150 165 180 195 210 P3Ph1 0,008 0,017 0,025 0,033 0,042 0,05 0,05 0,05 0,05 0,05 0,05 0,05 0,050 0,05 P3Ph2 - - - - 0,056 0,113 0,169 0,225 0,225 0,225 0,225 0,225 0,225 0,225 P3Ph3 - - - - - - 0,041 0,083 0,124 0,165 0,165 0,165 0,165 0,165 P3Ph4 - - - - - - - - 0,133 0,265 0,265 0,265 0,265 0,265 P3Ph5 - - - - - - - - 0,059 0,118 0,176 0,235 0,235 0,235 P3Ph6 - - - - - - - - - - - - 0,03 0,06 P QW P t0,008 0,017 0,025 0,033 0,098 0,163 0,26 0,358 0,59 0,823 0,881 0,94 0,97 1 Table 4.20: Planned contribution of the work performed vii Quality Metrics Review (See section iii.ii) For Project 3, 3 control points were defined to review the project quality metrics. The activities were carried out within the scheduled time, and there were no deviations from the plan. Therefore, we will show the evaluation of each project task in the Actual Contribution of Work Performed (ACWP) tables (See section 4.1.8.3). The evaluation was carried out from 0 to 1, with 1 being the optimal result of the task in terms of quality. 4.1. SUMMARY AND DISCUSSION. 109 The following ratings were obtained for the first phase of the project: P3P h1.1ϕj P3P h1.2ϕj P3P h1.3ϕj P3P h1.4ϕjWeighted sum tcj0,25 0,25 0,25 0,25 15 P3Ph1 - 1.1.1 0,167 0,9 - - - - - - 0,038 P3Ph1 - 1.1.2 0,167 0,8 - - - - - - 0,033 P3Ph1 - 1.1.3 0,167 0,8 - - - - - - 0,033 30 P3Ph1 - 1.1.4 0,167 0,9 0,111 0,7 - - - - 0,057 P3Ph1 - 1.1.5 0,167 1 - - - - - - 0,042 45 P3Ph1 - 1.1.6 0,167 0,9 0,111 1 - - - - 0,065 P3Ph1 - 1.2.1 - - 0,111 0,8 - - - - 0,022 IQT2.2 - 1.2.2 - - 0,111 1 0,25 0,5 - - 0,059 60 IQT2.2 - 1.2.3 - - 0,111 0,8 - - - - 0,022 P3Ph1 - 1.2.4 - - 0,111 0,9 - - - - 0,025 P3Ph1 - 1.2.5 - - 0,111 0,8 - - - - 0,022 P3Ph1 - 1.2.6 - - 0,111 1 - - - - 0,028 75 IQT2.2 - 1.2.7 - - 0,111 0,9 - - - - 0,025 P3Ph1 - 1.3.1 - - - - 0,25 1 - - 0,063 90 IQT2.2 - 1.3.2 - - - - 0,25 1 - - 0,063 IQT2.2 - 1.3.3 - - - - 0,25 0,8 - - 0,050 P3Ph1 - 1.4.1 - - - - - - 1 0,9 0,225 Table 4.21: Actual contribution of work performed - Phase 1 110 CHAPTER 4. SUMMARY AND DISCUSSION OF THE RESULTS. The information collected in Table 4.21, can be summarised as follows: t P3QC2 (0,050) Weighted sum Pjwjϕj(ˆxj)ˆr1j(t) r11 = 1,0 rcp 11(t)ϕ1(ˆx1) 15 0,167 0,1042 0,001 30 0,333 0,2028 0,003 45 0,5 0,3493 0,009 60 0,667 0,4465 0,015 75 0,833 0,534 0,022 90 1 0,8715 0,044 Table 4.22: ACWP - P3Ph1 Following the same evaluation method, assessments were made of the activities carried out in the subsequent phases of the project. Actual contribution of work performed - P3Ph2 t P3QC1 (0,150) r22 = 0,700 P3QC6 (0,150) r23 = 0,200 P3QC4 (0,200) r25 = 0,150 P3QC5 (0,300) r26 = 0,200 Weighted sum Pjwjϕj(ˆxj) ˆr2j(t) rcp 22(t)ϕ2(ˆx2)rcp 23(t)ϕ3(ˆx3)rcp 25(t)ϕ5(ˆx5)rcp 26(t)ϕ6(ˆx6) 75 0,175 0,225 0,050 0,225 0,038 0,225 0,050 0,225 0,013 90 0,350 0,442 0,100 0,442 0,075 0,442 0,100 0,442 0,050 105 0,525 0,742 0,150 0,742 0,113 0,742 0,150 0,742 0,125 120 0,700 0,892 0,200 0,892 0,150 0,892 0,200 0,892 0,201 Table 4.23: ACWP - P3Ph2 Actual contribution of work performed - P3Ph3 t P3QC6 (0,150) r33 = 0,500 P3QC4 (0,200) r35 = 0,150 P3QC5 (0,300) r36 = 0,200 Weighted sum Pjwjϕj(ˆxj) ˆr3j(t) rcp 33 (t)ϕ3(ˆx3)rcp 35 (t)ϕ5(ˆx5)rcp 36 (t)ϕ6(ˆx6) 105 0,125 0,136 0,038 0,136 0,050 0,136 0,006 120 0,250 0,314 0,075 0,314 0,100 0,314 0,026 135 0,375 0,633 0,113 0,633 0,150 0,633 0,078 150 0,500 0,893 0,150 0,893 0,200 0,893 0,147 Table 4.24: ACWP - P3Ph3 4.1. SUMMARY AND DISCUSSION. 111 Actual contribution of work performed - P3Ph4 t P3QC6 (0,150) r43 = 0,200 P3QC3 (0,150) r44 = 0,500 P3QC4 (0,200) r45 = 0,500 P3QC5 (0,300) r46 = 0,200 Weighted sum Pjwjϕj(ˆxj) ˆr4j(t) rcp 43 (t)ϕ3(ˆx3)rcp 44 (t)ϕ4(ˆx4)rcp 45 (t)ϕ5(ˆx5)rcp 46 (t)ϕ6(ˆx6) 135 0,100 0,467 0,250 0,467 0,250 0,467 0,100 0,467 0,062 150 0,200 0,917 0,500 0,917 0,500 0,917 0,200 0,917 0,243 Table 4.25: ACWP - P3Ph4 Actual contribution of work performed - P3Ph5 t P3QC1 (0,150) r52 = 0,300 P3QC6 (0,150) r53 = 0,100 P3QC3 (0,150) r54 = 0,500 P3QC4 (0,200) r55 = 0,200 P3QC5 (0,300) r56 = 0,200 Weighted sum Pjwjϕj(ˆxj)ˆr5j( t) rcp 52(t)ϕ2(ˆx2)rcp 53(t)ϕ3(ˆx3)rcp 54(t)ϕ4(ˆx4)rcp 55(t)ϕ5(ˆx5)rcp 56(t)ϕ6(ˆx6) 135 0,075 0,425 0,025 0,425 0,125 0,425 0,050 0,425 0,050 0,425 0,025 150 0,150 0,531 0,050 0,531 0,250 0,531 0,100 0,531 0,100 0,531 0,062 165 0,225 0,750 0,075 0,750 0,375 0,750 0,150 0,750 0,150 0,750 0,132 180 0,300 0,850 0,100 0,850 0,500 0,850 0,200 0,850 0,200 0,850 0,200 Table 4.26: ACWP - P3Ph5 Actual contribution of work performed - P3Ph6 t P3QC5 0,300 r66 = 0,200 Weighted sum Pjwjϕj(ˆxj)ˆr6j( t) rcp 66(t)ϕ6(ˆx6) 195 0,100 0,52 0,016 210 0,200 0,86 0,052 Table 4.27: ACWP - P3Ph6 118 References [36] H. Etzkowitz and L. Leydesdorff, “The dynamics of innovation: from national systems and “mode 2” to a triple helix of university–industry–government relations,” Research Policy, vol. 29, no. 2, pp. 109–123, 2000. [37] M. Gibbons, The new production of knowledge: The dynamics of science and research in contemporary societies. Sage, 1994. [38] E. Commission, “Improving knowledge transfer between research institutions and industry across europe,” tech. rep., European Commission, 2007. [39] A. Spithoven, B. Clarysse, and M. Knockaert, “Building absorptive capacity to organise inbound open innovation in traditional industries,” Technovation, vol. 30, no. 2, pp. 130–141, 2010. [40] U. AENOR, “Une 166001: 2006: Gesti´on de la i+ d+ i: Requisitos de un proyecto de i+d+i,” Norma espa˜nola, 2006. [41] R. Sternberg, “Reasons for the genesis of high-tech regions—theoretical explanation and empirical evidence,” Geoforum, vol. 27, no. 2, pp. 205–223, 1996. [42] J. WALLIN, O. ISAKSSON, A. LARSSON, and B.-O. ELFSTR¨ OM, “Bridging the gap between university and industry: Three mechanisms for innovation efficiency,” International Journal of Innovation and Technology Management, vol. 11, no. 01, p. 1440005, 2014. [43] R. Landry, N. Amara, J.-S. Cloutier, and N. Halilem, “Technology transfer organizations: Services and business models,” Technovation, vol. 33, no. 12, pp. 431–449, 2013. [44] J. Howells, “Intermediation and the role of intermediaries in innovation,” Research Policy, vol. 35, no. 5, pp. 715–728, 2006. [45] M. Agogu´e, E. Berthet, T. Fredberg, P. Le Masson, B. Segrestin, M. Stoetzel, M. Wiener, and A. Ystr¨om, “Explicating the role of innovation intermediaries in the “unknown”: a contingency approach,” Journal of Strategy and Management, 2017. [46] H. Chesbrough, W. Vanhaverbeke, T. Bakici, and H. Lopez-Vega, “Open innovation and public policy in europe,” 2011. [47] A. Arundel and A. Geuna, “Proximity and the use of public science by innovative european firms,” Economics of Innovation and New Technology, vol. 13, no. 6, pp. 559–580, 2004. [48] G. Schiuma, D. Carlucci, A. Lerro, R. Landry, and N. Amara, “Elucidation and enhancement of knowledge and technology transfer business models,” Vine, 2012. [49] Y. S. Lee, “The sustainability of university-industry research collaboration: An empirical assessment,” The journal of Technology transfer, vol. 25, no. 2, pp. 111– 133, 2000. [50] P. D’este and M. Perkmann, “Why do academics engage with industry? the entrepreneurial university and individual motivations,” The Journal of Technology Transfer, vol. 36, no. 3, pp. 316–339, 2011. References 119 [51] R. Stones, “You can’t always get what you want,” Let It Bleed, 1969. [52] M. Santoro and P. Bierly, “Facilitators of knowledge transfer in university-industry collaborations: A knowledge-based perspective,” IEEE Transactions on Engineering Management, vol. 53, no. 4, pp. 495–507, 2006. [53] T. Nomakuchi and M. Takahashi, “A study about project management for industry-university cooperation dilemma,” Procedia Computer Science, vol. 64, pp. 47–54, 2015. Conference on ENTERprise Information Systems/International Conference on Project MANagement/Conference on Health and Social Care Information Systems and Technologies, CENTERIS/ProjMAN / HCist 2015 October 7-9, 2015. [54] O. Hauptman and K. K. Hirji, “Managing integration and coordination in crossfunctional teams: an international study of concurrent engineering product development,” R&D Management, vol. 29, no. 2, pp. 179–192, 1999. [55] V. Morandi, “The management of industry–university joint research projects: how do partners coordinate and control r&d activities?,” The Journal of Technology Transfer, vol. 38, no. 2, pp. 69–92, 2013. [56] A. H. V. D. Ven, A. L. Delbecq, and R. Koenig, “Determinants of coordination modes within organizations,” American Sociological Review, vol. 41, no. 2, pp. 322– 338, 1976. [57] W. H. A. Johnson and D. A. Johnston, “Organisational knowledge creating processes and the performance of university-industry collaborative r&d projects,” Int. J. Technol. Manag., vol. 27, pp. 93–114, 2004. [58] S. Kobarg, J. Stumpf-Wollersheim, and I. M. Welpe, “University-industry collaborations and product innovation performance: The moderating effects of absorptive capacity and innovation competencies,” The Journal of Technology Transfer, vol. 43, no. 6, pp. 1696–1724, 2018. [59] S. U. Nsanzumuhire and W. Groot, “Context perspective on university-industry collaboration processes: A systematic review of literature,” Journal of Cleaner Production, vol. 258, p. 120861, 2020. [60] A. L. Oliver, K. Montgomery, and S. Barda, “The multi-level process of trust and learning in university–industry innovation collaborations,” The Journal of Technology Transfer, vol. 45, no. 3, pp. 758–779, 2020. [61] E. Bellini, G. Piroli, and L. Pennacchio, “Collaborative know-how and trust in university–industry collaborations: Empirical evidence from ict firms,” The Journal of Technology Transfer, vol. 44, no. 6, pp. 1939–1963, 2019. [62] E. Albats, I. Fiegenbaum, and J. A. Cunningham, “A micro level study of university industry collaborative lifecycle key performance indicators,” The Journal of Technology Transfer, vol. 43, no. 2, pp. 389–431, 2018. [63] C. Wolff and A. Nuseibah, “A projectized path towards an effective industryuniversity-cluster: Ruhrvalley,” in 2017 12th International Scientific and Technical Conference on Computer Sciences and Information Technologies (CSIT), vol. 2, 120 References pp. 123–131, 2017. [64] J. vom Brocke and S. Lippe, “Managing collaborative research projects: A synthesis of project management literature and directives for future research,” International Journal of Project Management, vol. 33, no. 5, pp. 1022–1039, 2015. [65] S. Ankrah and O. AL-Tabbaa, “Universities–industry collaboration: A systematic review,” Scandinavian Journal of Management, vol. 31, no. 3, pp. 387–408, 2015. [66] M. L. Mosbrooker, The implementation of project management: the professional’s handbook. Basic Books, 1981. [67] A. Gemino, B. H. Reich, and P. M. Serrador, “Agile, traditional, and hybrid approaches to project success: Is hybrid a poor second choice?,” Project Management Journal, vol. 52, no. 2, pp. 161–175, 2021. [68] A. L. Olechowski, S. D. Eppinger, N. Joglekar, and K. Tomaschek, “Technology readiness levels: Shortcomings and improvement opportunities,” Systems Engineering, vol. 23, no. 4, pp. 395–408, 2020. [69] ˚ Asa Fast-Berglund, L.-O. Blig˚ard, M. ˚ Akerman, and M. Karlsson, “Using the trlmethodology to design supporting ict-tools for production operators,” Procedia CIRP, vol. 17, pp. 726–731, 2014. Variety Management in Manufacturing. [70] N. Becheikh, R. Landry, and N. Amara, “Lessons from innovation empirical studies in the manufacturing sector: A systematic review of the literature from 1993–2003,” Technovation, vol. 26, no. 5, pp. 644–664, 2006. [71] J. C. Mankins, “Technology readiness assessments: A retrospective,” Acta Astronautica, vol. 65, no. 9, pp. 1216–1223, 2009. [72] B. J. Sauser, J. E. Ramirez-Marquez, D. Henry, and D. DiMarzio, “A system maturity index for the systems engineering life cycle,” International Journal of Industrial and Systems Engineering, vol. 3, no. 6, pp. 673–691, 2008. [73] J. C. Mankins, “Approaches to strategic research and technology (r&t) analysis and road mapping,” Acta Astronautica, vol. 51, no. 1, pp. 3–21, 2002. [74] J. Smith, “An alternative to technology readiness levels for non-developmental item (ndi) software,” in Proceedings of the 38th Annual Hawaii International Conference on System Sciences, pp. 315a–315a, 2005. [75] J. C. Mankins, “Technology readiness levels,” White Paper, April, vol. 6, p. 1995, 1995. [76] R. Van Dierdonck, K. Debackere, and B. Engelen, “University-industry relationships: How does the belgian academic community feel about it?,” Research Policy, vol. 19, no. 6, pp. 551–566, 1990. [77] E. Piva and C. Rossi-Lamastra, “Systems of indicators to evaluate the performance of university-industry alliances: a review of the literature and directions for future research,” Measuring Business Excellence, 2013. [78] M. Perkmann, A. Neely, and K. Walsh, “How should firms evaluate success in university–industry alliances? a performance measurement system,” R&D Man- References 121 agement, vol. 41, no. 2, pp. 202–216, 2011. [79] R. Magnaye, B. Sauser, P. Patanakul, D. Nowicki, and W. Randall, “Earned readiness management for scheduling, monitoring and evaluating the development of complex product systems,” International Journal of Project Management, vol. 32, no. 7, pp. 1246–1259, 2014. [80] K. Smith, “Measuring Innovation,” in The Oxford Handbook of Innovation, Oxford University Press, 01 2006. [81] T. R. Browning and R. V. Ramasesh, “A survey of activity network-based process models for managing product development projects,” Production and Operations Management, vol. 16, no. 2, pp. 217–240, 2007. [82] J. R. Hauser, D. Clausing, et al., “The house of quality,” Harvard Business Review, 1988. [83] S. Pugh, Total design: integrated methods for successful product engineering. Addison-Wesley, 1991. [84] J. E. Kasser, “The first requirements elucidator demonstration (fred) tool,” Systems Engineering, vol. 7, no. 3, pp. 243–256, 2004. [85] A. Vollerthun, “Design-to-market integrating conceptual design and marketing,” Systems Engineering, vol. 5, no. 4, pp. 315–326, 2002. [86] T. Shell, “The synthesis of optimal systems design solutions,” Systems Engineering, vol. 6, no. 2, pp. 92–105, 2003. [87] D. E. Whitney et al., “Manufacturing by design,” Harvard Business Review, vol. 66, no. 4, pp. 83–91, 1988. [88] K. B. Clark, “Product development performance: Strategy,” Organization, and Management in the World Auto Industry, 1991. [89] S. C. Wheelwright and K. B. Clark, Revolutionizing product development: quantum leaps in speed, efficiency, and quality. Simon and Schuster, 1992. [90] J. Brady, “Systems engineering and cost as an independent variable,” Systems Engineering, vol. 4, no. 4, pp. 233–241, 2001. [91] B. W. Oppenheim, “Lean product development flow,” Systems Engineering, vol. 7, no. 4, 2004. [92] K. S. Pawar and H. Driva, “Performance measurement for product design and development in a manufacturing environment,” International Journal of Production Economics, vol. 60-61, pp. 61–68, 1999. [93] A. Tiwana and M. Keil, “Control in internal and outsourced software projects,” Journal of Management Information Systems, vol. 26, no. 3, pp. 9–44, 2009. [94] R. G. Cooper and E. J. Kleinschmidt, “Winning businesses in product development: The critical success factors,” Research-Technology Management, vol. 50, no. 3, pp. 52–66, 2007. 122 References [95] D. Dvir and T. Lechler, “Plans are nothing, changing plans is everything: the impact of changes on project success,” Research Policy, vol. 33, no. 1, pp. 1–15, 2004. [96] J. Pinto and S. Mantel, “The causes of project failure,” IEEE Transactions on Engineering Management, vol. 37, no. 4, pp. 269–276, 1990. [97] J. K. Pinto and D. P. Slevin, “Critical factors in successful project implementation,” IEEE Transactions on Engineering Management, vol. EM-34, no. 1, pp. 22– 27, 1987. [98] C. K. Bart, “Controlling new products: a contingency approach,” International Journal of Technology Management, vol. 18, no. 5-8, pp. 395–413, 1999. [99] H. Cortes, J. Daaboul, J. Le Duigou, and B. Eynard, “Strategic lean management: Integration of operational performance indicators for strategic lean management,” IFAC-PapersOnLine, vol. 49, no. 12, pp. 65–70, 2016. 8th IFAC Conference on Manufacturing Modelling, Management and Control MIM 2016. [100] S. ur Rehman Toor and S. O. Ogunlana, “Beyond the ‘iron triangle’: Stakeholder perception of key performance indicators (kpis) for large-scale public sector development projects,” International Journal of Project Management, vol. 28, no. 3, pp. 228–236, 2010. [101] H. Kerzner, Project management metrics, KPIs, and dashboards: a guide to measuring and monitoring project performance. John Wiley & Sons, 2022. [102] F. Espa˜na, C. C. Tsao, and M. Hauser, “Driving continuous improvement by developing and leveraging lean key performance indicators,” in annual conference of the international group for lean construction, vol. 20, 2012. [103] U. Dombrowski, K. Schmidtchen, and D. Ebentreich, “Balanced key performance indicators in product development,” International Journal of Materials, Mechanics and Manufacturing, vol. 1, no. 1, pp. 27–31, 2013. [104] F. A. Mir and A. H. Pinnington, “Exploring the value of project management: Linking project management performance and project success,” International Journal of Project Management, vol. 32, no. 2, pp. 202–217, 2014. [105] O. Bizan, “The determinants of success of r&d projects: evidence from american–israeli research alliances,” Research Policy, vol. 32, no. 9, pp. 1619–1640, 2003. [106] A. J. Shenhar, A. Tishler, D. Dvir, S. Lipovetsky, and T. Lechler, “Refining the search for project success factors: a multivariate, typological approach,” R&D Management, vol. 32, no. 2, pp. 111–126, 2002. [107] A. de Wit, “Measurement of project success,” International Journal of Project Management, vol. 6, no. 3, pp. 164–170, 1988. [108] T. Cooke-Davies, “The “real” success factors on projects,” International Journal of Project Management, vol. 20, no. 3, pp. 185–190, 2002. [109] Axelos, A Guide to AgileSHIFT. The Stationery Office, London, UK, 2018. References 123 [110] A. Bennett, Managing Successful Projects with PRINCE2. The Stationery Office, London, UK, 2017. [111] A. ul Musawir, C. E. M. Serra, O. Zwikael, and I. Ali, “Project governance, benefit management, and project success: Towards a framework for supporting organizational strategy implementation,” International Journal of Project Management, vol. 35, no. 8, pp. 1658–1672, 2017. [112] H. Kerzner, Project management: a systems approach to planning, scheduling, and controlling. John Wiley & Sons, 2017. [113] P. L. Bannerman, “Defining project success: a multilevel framework,” in Defining the Future of Project Management, (Warsaw, Poland), Project Management Institute, 2008. [114] G. Thomas and W. Fern´andez, “Success in it projects: A matter of definition?,” International Journal of Project Management, vol. 26, no. 7, pp. 733–742, 2008. Special Issue: Achieving IT Project Success. [115] Y. Xue, J. R. Turner, L. Lecoeuvre, and F. Anbari, “Using results-based monitoring and evaluation to deliver results on key infrastructure projects in china,” Global Business Perspectives, vol. 1, no. 2, pp. 85–105, 2013. [116] P. M. Institute, “uccess in disruptive times — pulse of the profession 2018,” tech. rep., Project Management Institute, 2018. [117] O. Pankratz and D. Basten, “Opening the black box: Managers’ perceptions of is project success mechanisms,” Information & Management, vol. 55, no. 3, pp. 381– 395, 2018. [118] W. Abba, “How earned value got to primetime: A short look back and a glance ahead,” in Project management institute seminars and symposium in Houston, TX, 2000. [119] S. Rozenes, G. Vitner, and S. Spraggett, “Mpcs: Multidimensional project control system,” International Journal of Project Management, vol. 22, no. 2, pp. 109–118, 2004. [120] M. Lauras, G. Marques, and D. Gourc, “Towards a multi-dimensional project performance measurement system,” Decision Support Systems, vol. 48, no. 2, pp. 342– 353, 2010. [121] ¨ Onc¨u Hazır, “A review of analytical models, approaches and decision support tools in project monitoring and control,” International Journal of Project Management, vol. 33, no. 4, pp. 808–815, 2015. [122] P. J. Solomon and R. R. Young, Performance-based earned value. Citeseer, 2007. [123] B. Pollack-Johnson and M. Liberatore, “Incorporating quality considerations into project time/cost tradeoff analysis and decision making,” IEEE Transactions on Engineering Management, vol. 53, no. 4, pp. 534–542, 2006. [124] M. J. Liberatore and B. Pollack-Johnson, “Improving project management decision making by modeling quality, time, and cost continuously,” IEEE Transactions on 124 References Engineering Management, vol. 60, no. 3, pp. 518–528, 2013. [125] M. A. Bragadin and K. K¨ahk¨onen, “Schedule health assessment of construction projects,” Construction Management and Economics, vol. 34, no. 12, pp. 875–897, 2016. [126] J. Pitk¨anen et al., “Project monitoring in industrial mechanical installations,” Master’s thesis, Aalto University, 2017. [127] E. Akg¨un, “Quality integrated earned value management for construction projects,” Master’s thesis, Middle East Technical University, 2019. [128] H. Jing, O. Hisarciklilar, and V. Thomson, “Ensuring correct product characteristics during the design process,” in Proceedings of the 2014 International Conference on Innovative Design and Manufacturing (ICIDM), pp. 193–198, IEEE, 2014. [129] G. Schuh, M. Riesener, C. Doelle, and S. Brockmann, “Evaluation of the effects of activity deviations onto a development project’s target dimensions,” in 2017 Portland International Conference on Management of Engineering and Technology (PICMET), pp. 1–6, IEEE, 2017. [130] T. Carbone, “Integrating operations and product development methodologies for improved product success using advanced product quality planning,” in IEEE/SEMI Conference and Workshop on Advanced Semiconductor Manufacturing 2005., pp. 228–233, 2005. [131] J. Kim, C. Kang, and I. Hwang, “A practical approach to project scheduling: considering the potential quality loss cost in the time–cost tradeoff problem,” International Journal of Project Management, vol. 30, no. 2, pp. 264–272, 2012. [132] E. Commission and D.-G. for Informatics, PM², Project management methodology guide : open edition. Publications Office of the European Union, 2016. [133] P. M. Institute et al.,A Guide to the Project Management Body of Knowledge: PMBOK Guide. Project Management Institute, Inc., 2021. [134] R. G. Cooper, “Third-generation new product processes,” Journal of Product Innovation Management, vol. 11, no. 1, pp. 3–14, 1994. [135] R. G. Cooper, “Stage-gate systems: a new tool for managing new products,” Business horizons, vol. 33, no. 3, pp. 44–54, 1990. [136] H. Kerzner, Strategic planning for project management using a project management maturity model. John Wiley & Sons, 2002. [137] M. L. Drury-Grogan, “Performance on agile teams: Relating iteration objectives and critical decisions to project management success factors,” Information and Software Technology, vol. 56, no. 5, pp. 506–515, 2014. Performance in Software Development. [138] R. Guti´errez, J. Canela, T. Femen´ıas, and F. Art´es, “Experiences in agile r&d project management for new product design and development in the automotive industry,” J. Trends Dev. Mach. Assoc. Technol, vol. 16, pp. 83–86, 2012. References 125 [139] R. G. Cooper, “What’s next?: After stage-gate,” Research-Technology Management, vol. 57, no. 1, pp. 20–31, 2014. [140] N. D. du Preez and L. Louw, “A framework for managing the innovation process,” in PICMET ’08 - 2008 Portland International Conference on Management of Engineering & Technology, pp. 546–558, 2008. [141] A. F. Sommer, C. Hedegaard, I. Dukovska-Popovska, and K. Steger-Jensen, “Improved product development performance through agile/stage-gate hybrids: The next-generation stage-gate process?,” Research-Technology Management, vol. 58, no. 1, pp. 34–45, 2015. [142] E. Fijn, “Innovation projects and their promising project management approach,” Master’s thesis, TU Delft, 2016. [143] H. Werner, Strategisches Forschungs-und Entwicklungs-Controlling. SpringerVerlag, 2013. [144] J. E. Riedl, Projekt-Controlling in Forschung und Entwicklung: Grunds¨atze, Methoden, Verfahren, Anwendungsbeispiele aus der Nachrichtentechnik. SpringerVerlag, 2013. [145] L. Sastoque Pinilla, B. J. Ngereja, N. Mikhridinova, C. Wolff, and N. Toledo Gandarias, “Knowledge discovery process applied to building competence profiles description,” in Dortmund International Research Conference 2022 Proceedings, Fachhochschule Dortmund; Dortmund, 2022. [146] D. H. Meadows, Thinking in systems: A primer. chelsea green publishing, 2008. [147] S. Marcelino-S´adaba, A. P´erez-Ezcurdia, A. M. Echeverr´ıa Lazcano, and P. Villanueva, “Project risk management methodology for small firms,” International Journal of Project Management, vol. 32, no. 2, pp. 327–340, 2014. [148] M. Fowler, J. Highsmith, et al., “The agile manifesto,” Software development, vol. 9, no. 8, pp. 28–35, 2001. [149] S. Artelt, “Mitigate the risk of project schedule overruns and project delay chains in a trl 5-7 research centre,” Master’s thesis, University of the Basque Country, 2020. [150] H. Kerzner, Innovation project management: Methods, case studies, and tools for managing innovation projects. John Wiley & Sons, 2022. [151] L. Argote, “Input uncertainty and organizational coordination in hospital emergency units,” Administrative Science Quarterly, vol. 27, no. 3, pp. 420–434, 1982. [152] H. Sicotte and A. Langley, “Integration mechanisms and r&d project performance,” Journal of Engineering and Technology Management, vol. 17, no. 1, pp. 1–37, 2000. [153] S. Ward and C. Chapman, “Stakeholders and uncertainty management in projects,” Construction Management and Economics, vol. 26, no. 6, pp. 563–577, 2008. [154] J. Charvat, Project management methodologies: selecting, implementing, and supporting methodologies and processes for projects. John Wiley & Sons, 2003. 126 References [155] C. Chin, E. Yap, A. Spowage, et al., “Project management methodology for university-industry collaborative projects,” Review of International Comparative Management, vol. 12, no. 5, pp. 901–918, 2011. [156] “Bolet´ın oficial del pa´ıs vasco - bopv n.º30, viernes 11 de febrero de 2022,” 2022. [157] S. Kabadayi, A. Pridgen, and C. Julien, “Virtual sensors: abstracting data from physical sensors,” in 2006 International Symposium on a World of Wireless, Mobile and Multimedia Networks(WoWMoM’06), pp. 6 pp.–592, 2006. [158] T. L. Saaty, “How to make a decision: The analytic hierarchy process,” European Journal of Operational Research, vol. 48, no. 1, pp. 9–26, 1990. Desicion making by the analytic hierarchy process: Theory and applications. [159] A. Rodr´ıguez, A. Fern´andez, L. N. L´opez de Lacalle, and L. Sastoque Pinilla, “Flexible abrasive tools for the deburring and finishing of holes in superalloys,” Journal of Manufacturing and Materials Processing, vol. 2, no. 4, 2018. [160] G. G. Escudero, H. Gonz´alez, A. Calleja, L. S. Pinilla, and I. A. Rementeria, “Tecnolog´ıas clave para la nueva f´abrica inteligente,” Eurofach electronica: Actualidad y tecnolog´ıa de la industria electr´onica, no. 475, pp. 28–36, 2020. [161] L. L´opez de Lacalle, A. Fern´andez Valdivielso, F. Amigo, and L. Sastoque, “Milling with ceramic inserts of austempered ductile iron (adi): process conditions and performance,” The International Journal of Advanced Manufacturing Technology, vol. 110, no. 3, pp. 899–907, 2020. [162] A. D. Olmo, G. M. de Piss´on, L. Sastoque, A. Fern´andez, A. Calleja, and L. N. L. D. Lacalle, “Merging complex information in high speed broaching operations in order to obtain a robust machining process,” IOP Conference Series: Materials Science and Engineering, vol. 1193, oct 2021. [163] N. Mikhridinova, B. J. Ngereja, L. Sastoque Pinilla, C. Wolff, and W. Van Petegem, “Developing and improving competence profiles of project teams in engineering education,” in SEFI Annual Conference, 2022. Part II Conclusions 230 CHAPTER 10. COMMUNICATIONS AND OTHER PUBLICATIONS. 10.2 Other publications. •Original Journal Paper. 1. Rodr´ıguez, A., Fern´andez, A., L´opez de Lacalle, L. N., & Sastoque Pinilla, L. (2018). Flexible abrasive tools for the deburring and finishing of holes in superalloys. Journal of Manufacturing and Materials Processing, 2(4), 82 [159]. 2. Escudero, G. G., Gonz´alez, H., Calleja, A., Pinilla, L. S., & Rementeria, I. A. (2020). Tecnolog´ıas clave para la nueva f´abrica inteligente. Eurofach electronica: Actualidad y tecnolog´ıa de la industria electr´onica, (475), 28-36 [160]. 3. L´opez de Lacalle, L. N., Fern´andez Valdivielso, A., Amigo, F. J., & Sastoque, L. (2020). Milling with ceramic inserts of austempered ductile iron (ADI): process conditions and performance. The International Journal of Advanced Manufacturing Technology, 110(3), 899-907 [161]. 4. Del Olmo, A., de Lacalle, L. L., de Piss´on, G. M., P´erez-Salinas, C., Ealo, J. A., Sastoque, L., & Fernandes, M. H. (2022). Tool wear monitoring of high-speed broaching process with carbide tools to reduce production errors. Mechanical Systems and Signal Processing, 172, 109003 [13]. •Conference Paper. 1. Del Olmo, A., de Piss´on, G. M., Sastoque, L., Fern´andez, A., Calleja, A., & De Lacalle, L. L. (2021, October). Merging complex information in high-speed broaching operations in order to obtain a robust machining process. In IOP Conference Series: Materials Science and Engineering (Vol. 1193, No. 1, p. 012079). IOP Publishing [162]. 2. Sastoque-Pinilla, L., Mikhridinova, N., Ngereja, B. J., Wolff, C., & Toledo Gandarias, N. (2022). Knowledge Discovery Process Applied to Building Competence Profiles Description. In Dortmund International Research Conference 2022 Proceedings. Fachhochschule Dortmund; Dortmund [145]. 3. Mikhridinova, N., Ngereja, B. J., Sastoque Pinilla, L., Wolff, C., & Van Petegem, W. (2022). Developing and improving competence profiles of project teams in engineering education. In SEFI Annual Conference [163]. 11 Data Appendix. 11.1 CFAA’s KPIs. 232 CHAPTER 11. DATA APPENDIX. 1. Scope - 1.1. Strategical Need - 1.1.1. Operational Need / Category Financial Life cycle Operational Project Strategic Sustainability Technical Total 1. Innovation — — 7 — 5 12 10 34 1.1. Creating and marketing new services, product lines, and technological capabilities. — — 7 — 2 12 3 24 1.1.1. Contribute to sustainability in the creation, design and result of deliverables from the projects — — — — — 12 — 12 1.1.2. Develop and implement marketing of the R&D service portfolio — — — — — — 1 1 1.1.3. Generate new usable knowledge and engineering solutions — — 7 — — — — 7 1.1.4. Increase demand orientation in transfer — — — — 2 — — 2 1.1.5. Introduce continuous innovation management — — — — — — 2 2 1.2. Institutional support of SMEs with innovation impulses in developing new business models. — — — — 3 — 7 10 1.2.1. Enable and encourage talent early on — — — — — — 3 3 1.2.2. Expand existing cooperation into strategic innovation partnerships — — — — 3 — — 3 1.2.3. Promote the foundation, establishment and accompaniment of spinoffs — — — — — — 4 4 2. Region 6 — 5 5 18 — 4 38 11.1. CFAA’S KPIS. 233 2.1. CFAA is a place of interest for local and international partners to develop projects. 6 — 4 5 3 — 1 19 2.1.1 Assure by any possible means that the project’s results are helpful for the project partner — — 2 — — — — 2 2.1.2 Assure fluid communication with the partner before, during and after the development of the project — — — — 2 — 1 3 2.1.4 Find partners eager to share and collaborate with knowledge, commercial and technical interests. — — — — 1 — — 1 2.1.5 Increase the reputation of PMO — — — 5 — — — 5 2.1.6 Increase the return of investment of the projects 6 — — — — — — 6 2.1.7 Increasing awareness of CFAA as a centre of excellence — — 1 — — — — 1 2.1.8 Project development according to the partners’ expectations — — 1 — — — — 1 2.2. Increase the attractiveness of the Basque Country for research, innovation, employment and start-ups — — — — 9 — — 9 2.2.1 Contribute to the creation of wealth in the Basque Country, supporting and promoting sustainable competitiveness of Basque Country Companies — — — — 4 — — 4 2.2.2 Involve local companies of strategical sectors in R&D projects — — — — 1 — — 1 234 CHAPTER 11. DATA APPENDIX. 2.2.3 Promote the internationalisation and technological innovation of partners companies — — — — 4 — — 4 2.3. Intensify knowledge transfer — — 1 — 6 — — 7 2.3.1 Contribute to the advancement of knowledge and social development through R&D and innovation — — — — 4 — — 4 2.3.2 Increase graduate retention in SMEs in the region — — — — 1 — — 1 2.3.3 Involve personnel from partner companies as active actors in the implementation and development of projects — — 1 — — — — 1 2.3.4 Promote and coordinate the technology transfer of R&D and Innovation activities — — — — 1 — — 1 2.4. Interact with society, strengthen actors and civil society — — — — — — 3 3 2.4.1 Presence at popular science events — — — — — — 1 1 2.4.2 Presence in regional and national press — — — — — — 1 1 2.4.3 Presence on relevant portals and social media — — — — — — 1 1 3. Research — — — — 6 — 4 10 11.1. CFAA’S KPIS. 235 3.1. Gather and generate experience, new knowledge and understanding from activities and management of the project (e.g. tools, techniques, approaches or processes) — — — — 4 — 2 6 3.1.1 Develop, test, establish and transfer theory and methodology. — — — — 4 — — 4 3.1.2 Issuing central reference publications — — — — — — 2 2 3.2. Increase the impact of R&D in the scientific community — — — — 2 — 2 4 3.2.1 Increase publication output and conference attendances — — — — — — 1 1 3.2.2 Initiation of joint projects with renowned research partners — — — — 2 — — 2 3.2.3 Scientific publications from project results in refereed journals — — — — — — 1 1 4. University 1 5 2 6 12 — 16 42 4.1. Professionalizing project management — — — 6 3 — 1 10 4.1.1 Implement troubleshooting methods and process — — — — — — 1 1 4.1.2 Performance monitoring and evaluation of projects development and results — — — 5 — — — 5 4.1.3 Risk evaluation of realised opportunities (known unknowns) and suffered threats — — — 1 — — — 1 236 CHAPTER 11. DATA APPENDIX. 4.1.4 Set up strategic and operational multi-project management — — — — 3 — — 3 4.2. Promote compliance with the goals and objectives for which CFAA was created. 1 — — — 1 — 10 12 4.2.1 Ensure the financial, scientific and international success of the CFAA and the continuity of the activities 1 — — — — — 7 8 4.2.2 Meet the technical and management objectives of the Centre — — — — — — 2 2 4.2.3 Strategic potential of the projects developed — — — — 1 — — 1 4.2.4 Top management support to assist the compliment of the mission, policies and strategies of CFAA — — — — — — 1 1 4.3. Securing the freedom, financial and personnel basis for research and transfer activities in the long term — 5 1 — 3 — 5 14 4.3.1 Creating scope for research — — — — 2 — 4 6 4.3.2 Detect, analyse and resolve non conformities at time — 2 — — — — — 2 4.3.3 Keep the technical capacities needed for the project’s development. — 3 — — — — — 3 4.3.4 Mobilisation of resources for projects as needed — — 1 — — — — 1 4.3.5 Participation in programmes initiatives of the EU Research Framework Programme — — — — 1 — — 1 11.1. CFAA’S KPIS. 237 4.3.6 Securing funding for management capacity — — — — — — 1 1 4.4. Strengthen collaboration with the University in the implementation and development of projects — — 1 — 5 — — 6 4.4.1 Derive teaching content from RDI projects’ results. — — — — 1 — — 1 4.4.2 Encourage the participation of students in the development of projects — — — — 1 — — 1 4.4.3 Organize, coordinate and direct the activities in the Centre — — 1 — — — — 1 4.4.4 Prepare students, company and university staff through specialized training — — — — 1 — — 1 4.4.5 Strengthening research-based and practice-oriented teaching — — — — 2 — — 2 Total general 7 5 14 11 41 12 34 124 Table 11.1: CFAA’s KPIs 238 CHAPTER 11. DATA APPENDIX. 11.2 Proposed databases and classification scheme. Database Minimum information to be included Notes Critical Success Factors Future potential Partner satisfaction Project goals and mission (knowledge generation) Project management success factors KPIs Historically used KPIs KPIs per partner KPIs per technology KPIs per type of project Lessons Learned Lessons learn report of the project Lessons learned per partner Lessons learned per technology Lessons learned per type of project Partners’ evaluation Partners’ evaluation of life cycle of the project Partners’ evaluation of the deliverable Partners’ evaluation per technology Partners’ evaluation per type of project Partners’ feedback about the future implementation of the project’s deliverable. Project Evaluation Milestones compliment Project success evaluation Qualitative evaluation Quantitative evaluation Project Team Evaluation Qualitative evaluation Quantitative evaluation based on data accumulated throughout the project. 11.2. PROPOSED DATABASES AND CLASSIFICATION SCHEME. 239 Projects information Activation date Deliverable list Due Date Financing entity Partners involved Partners participation Planned cost Planned hours Planned resources Prioritization Project’ technology Scope Total cost type of project We proposed to classify the projects in the following categories: i) Creation or development of new product technology platform projects; ii) Innovation projects, iii) Product or technology enhancement projects, iv) Project oriented to new product or process. Quality Evaluation KPIs evaluation Quality assessment of the tasks We proposed to organise the information about the quality assessment on the following categories: i) Type of the project, ii) Classification of the Activity, iii) Used resources Quality Control Activities Quality evaluation per partner Quality evaluation per technology Quality evaluation per type of project Risk Assessment Ongoing project risks identification Previously identified risks (evaluation in terms of probability and impact) Risk mitigation activities