European Common Evaluation Methodology Handbook for Connected and Automated Mobility (EU-CEM Handbook for CCAM)

Version 1.0 / 5 May 2025 2 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Document Control Sheet Version history: Legal disclaimer: The FAME project has received funding by the European Union’s Horizon Europe research and innovation programme HORIZON-CL5-2021-D6-01-06 under Grant Agreement 101069898. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible for them. Preferred reference FAME (2025). European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility (EU-CEM Handbook for CCAM). https://www.connectedautomateddriving.eu/methodology/common-evaluation-methodology/ Version 1.0 Date 5 May 2025 Status Final Dissemination level Public Authors Innamaa S, Aittoniemi E, Sintonen H, Barnard Y, Drossel P, Geissler T, Harrison G, Huschebeck M, Lehtonen E, Mahdavi H, Malin F, Morcelli S, Nieto M, O’Hern S, Rosenquist M, Tol E, Wilmink I, Zubin I Key words Evaluation, Impact Assessment, Methodology, CCAM, Handbook Original manuscript FAME Deliverable D4.2 Common Evaluation Methodology Handbook for CCAM (EU-CEM). Submitted to EC on March 3, 2025. Please note that the final version of the handbook has been revised from this original manuscript, with new design, reference style, and corrections of minor errors. Version Description May 2024 Draft published for public consultation May 2025 Final version 1.0 3 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table of Contents Table of Guidelines ................................................................................................................................................. 4 Table of Figures ...................................................................................................................................................... 6 Table of Tables ........................................................................................................................................................ 8 Abbreviations & Terms ......................................................................................................................................... 11 Executive Summary .............................................................................................................................................. 13 1. Introduction .................................................................................................................................................... 15 1.1. Background ............................................................................................................................................ 15 1.2. Scope of EU-CEM ................................................................................................................................... 17 1.3. Structure of the EU-CEM Handbook ...................................................................................................... 21 1.4. Compliance with EU-CEM ...................................................................................................................... 22 1.5. Outlook .................................................................................................................................................. 22 2. Guidelines for the project preparation phase ................................................................................................. 23 2.1. Scoping of evaluation ............................................................................................................................ 23 2.2. Project structure and governance ......................................................................................................... 27 3. Guidelines for developing an evaluation plan ................................................................................................ 35 3.1. CCAM system description ...................................................................................................................... 36 3.2. Research questions ................................................................................................................................ 43 3.3. Evaluation methods ............................................................................................................................... 52 3.4. Data specification and data tools .......................................................................................................... 63 3.5. Experimental design .............................................................................................................................. 70 3.6. Evaluation plan compilation .................................................................................................................. 81 4. Evaluation-area-specific guidelines................................................................................................................. 86 4.1. Vehicle ................................................................................................................................................... 87 4.2. Human ................................................................................................................................................. 105 4.3. Transport system ................................................................................................................................. 122 4.4. Society ................................................................................................................................................. 171 5. Guidelines for reporting evaluation outcomes and EU-CEM feedback ........................................................ 213 5.1. Evaluation reporting ............................................................................................................................ 213 5.2. Reporting lessons learned and applicability of EU-CEM guidelines ..................................................... 215 5.3. Further guidance.................................................................................................................................. 215 References .......................................................................................................................................................... 216 Appendix ............................................................................................................................................................. 224 4 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table of Guidelines Guideline 2.1. How to set the overall scope of the evaluation ..................................................................... 23 Guideline 2.2. How to choose impact areas for evaluation .......................................................................... 24 Guideline 2.3. How to build a consortium fit for the evaluation scope ........................................................ 26 Guideline 2.4. How to plan evaluation activities for the proposal ................................................................ 28 Guideline 2.5. How to create a project structure that supports evaluation ................................................. 29 Guideline 2.6. How to assign responsibilities across work packages and activities ...................................... 30 Guideline 2.7. How to ensure information exchange to support the evaluation of CCAM .......................... 31 Guideline 2.8. How to enable adaptivity of the evaluations ......................................................................... 32 Guideline 2.9. How to ensure the quality of the evaluation ......................................................................... 32 Guideline 2.10. How to set up a process for mediating conflicts .................................................................... 33 Guideline 3.1. How to describe the CCAM system ........................................................................................ 37 Guideline 3.2. How to describe the ODD of an automated driving system .................................................. 39 Guideline 3.3. How to describe the CCAM service concept .......................................................................... 40 Guideline 3.4. How to describe the gap between the CCAM system tested in field experiments and the system used for the impact assessment ............................................................................................................... 41 Guideline 3.5. How to elaborate the scope of evaluation ............................................................................. 44 Guideline 3.6. How to formulate and organise research questions .............................................................. 45 Guideline 3.7. How to prioritise research questions ..................................................................................... 47 Guideline 3.8. How to check the feasibility of research questions ............................................................... 49 Guideline 3.9. How to set hypotheses .......................................................................................................... 50 Guideline 3.10. How to describe the societal scenarios for impact assessment ............................................ 54 Guideline 3.11. How to define midand low-level scenarios: traffic and driving scenarios, user and user behaviour scenarios .............................................................................................................................................. 56 Guideline 3.12. How to define relevant indicators for impact assessment .................................................... 59 Guideline 3.13. How to set a method for answering a research question ...................................................... 60 Guideline 3.14. How to consider the scaling up of results in the methodology ............................................. 61 Guideline 3.15. How to consider intended direct impacts, indirect and unintended impacts, and the impacts on non-users 62 Guideline 3.16. How to set up a process for defining data specification and data tools ................................ 65 Guideline 3.17. How to define common data formats .................................................................................... 66 Guideline 3.18. How to define the roles and access rights related to data .................................................... 67 Guideline 3.19. How to define the data flow in the processing chain ............................................................ 68 Guideline 3.20. How to overcome common pitfalls related to data ............................................................... 69 Guideline 3.21. How to choose the experimental approach for a research question .................................... 71 Guideline 3.22. How to consider the elements of the experimental design in the evaluation plan ............... 72 Guideline 3.23. How to plan the experimental design of field experiments ................................................... 75 Guideline 3.24. How to plan the experimental design for virtual environments with participants ................ 77 Guideline 3.25. How to select participants for your experiment .................................................................... 78 Guideline 3.26. How to provide participants with the most realistic user experience possible ..................... 79 5 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Guideline 3.27. How to define the treatment and baseline conditions .......................................................... 80 Guideline 3.28. How to compile the evaluation plan ...................................................................................... 82 Guideline 3.29. How to describe the linkages between different evaluation and impact areas .................... 83 Guideline 3.30. How to ensure resilience of the evaluation plan ................................................................... 84 6 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table of Figures Figure 1. Components of the European framework for testing on public roads. ................................................. 15 Figure 2. Core principles of EU-CEM. .................................................................................................................... 18 Figure 3. Role of different chapters of the EU-CEM Handbook (light green) and their outputs (dark green)...... 21 Figure 4. Workflow and iterations in setting the evaluation plan until a feasible plan is established. ................ 36 Figure 5. Evaluation areas addressed in Chapter 4. ............................................................................................. 86 Figure 6. Main components of the technical functioning evaluation area. .......................................................... 90 Figure 7. Output indicators of technical functioning. ........................................................................................... 92 Figure 8. Main components of the driving behaviour impact area. ..................................................................... 97 Figure 9. Output indicators of driving behaviour. .............................................................................................. 100 Figure 10. Main components of the user evaluation area. ................................................................................ 106 Figure 11. Input needed from the other impact areas for user evaluation and its outputs. .............................. 108 Figure 12. Main components of the people mobility impact area. .................................................................... 112 Figure 13. Input needed from the other impact areas for people mobility impact assessment and its outputs. ............................................................................................................................................................................ 114 Figure 14. Main components of the quality-of-life impact area. ........................................................................ 118 Figure 15. Inputs needed from the other impact areas for the quality-of-life impact assessment. .................. 120 Figure 16. Main components of the services and operation impact area. ......................................................... 124 Figure 17. Input needed from the other impact areas for evaluation of services and operation and its outputs. ............................................................................................................................................................................ 126 Figure 18. Main components of the logistics impact assessment. ..................................................................... 131 Figure 19. Input needed from the other impact areas for logistics impact assessment and its outputs. .......... 133 Figure 20. Main components of the transport activity and fleet composition impact area............................... 136 Figure 21. Input needed from the other impact areas for transport activity and fleet composition impact assessment and its outputs. ............................................................................................................................... 138 Figure 22. Three dimensions of traffic safety (redrawn from Nilsson [58]). ...................................................... 142 Figure 23. Main components of the traffic safety impact area. ......................................................................... 142 Figure 24. Traffic safety pyramid (adapted from Hydén [62]). ........................................................................... 144 Figure 25. Input needed from the other impact areas for traffic safety impact assessment and its outputs. ... 145 Figure 26. Main components of the traffic flow efficiency impact area. ........................................................... 150 Figure 27. Input needed from the other impact areas for traffic flow efficiency impact assessment and its outputs. .............................................................................................................................................................. 152 Figure 28. Main components of the energy and environment impact area....................................................... 158 Figure 29. Input needed from the other impact areas for energy and environmental impacts of CCAM systems. ............................................................................................................................................................................ 160 Figure 30. Main components of the accessibility impact area. .......................................................................... 166 Figure 31. Input needed from the other impact areas for accessibility impact assessment and its outputs. .... 168 Figure 32. Land-use transport feedback cycle (redrawn from Wegener & Fuerst [90]) ..................................... 172 Figure 33. Main components of the land use impact area. ................................................................................ 173 Figure 34. Input needed from the other impact areas for land use impact assessment and its outputs. ......... 175 7 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 35. High-level summary of the causal loop diagram of CCAM impacts on liveability (redrawn from Harrison et al. [94]). ............................................................................................................................................ 179 Figure 36. Main components of the liveability impact area (redrawn from Anciaes & Jones [97]). .................. 180 Figure 37. Input needed from other impact areas for liveability impact assessment and its output. ............... 182 Figure 38. Main components of the economic activity and employment impact area. ..................................... 187 Figure 39. Input needed from the other impact areas for economic activity and employment impact assessment and its outputs. ............................................................................................................................... 189 Figure 40. Main components of the socio-economic impact area. .................................................................... 192 Figure 41. Input needed from the other impact areas for socio-economic impact assessment and its outputs. ............................................................................................................................................................................ 194 Figure 42. Equality, equity, and justice. .............................................................................................................. 198 Figure 43. Main components of the equity impact area. ................................................................................... 199 Figure 44. Input needed for the equity impact assessment and its output. ...................................................... 204 Figure 45. Three ways to illustrate three pillars of sustainability: Venn diagram, nested model and three pillars. ............................................................................................................................................................................ 208 8 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table of Tables Table 1. Evaluation and impact areas covered by the EU-CEM Handbook and their scope. ............................... 19 Table 2. Indicators recommended to be evaluated by every project addressing technical functioning. ............. 91 Table 3. An overview of approaches and methods for evaluation of technical functioning. ............................... 92 Table 4. Pitfalls in evaluation of technical functioning and best practices to avoid them. .................................. 94 Table 5. Indicators recommended to be evaluated by every project addressing driving behaviour. .................. 98 Table 6. Outputs from driving behaviour evaluation required as input for other impact areas. ....................... 100 Table 7. An overview of approaches and methods for evaluation of driving behaviour.................................... 101 Table 8. Pitfalls in the evaluation of driving behaviour and best practices to avoid them. ............................... 103 Table 9. Indicators recommended to be evaluated by every project addressing user evaluation. .................... 107 Table 10. Outputs from user evaluation required as input for other impact areas. .......................................... 108 Table 11. An overview of approaches and methods for user evaluation. .......................................................... 109 Table 12. Pitfalls in user evaluation and best practices to avoid them. ............................................................. 110 Table 13. Indicators recommended to be evaluated by every project addressing people mobility. ................. 113 Table 14. Outputs from people mobility impact assessment required as input for other impact areas. .......... 114 Table 15. An overview of approaches and methods for people mobility impact assessment. .......................... 115 Table 16. Pitfalls in people mobility impact assessment and best practices to avoid them. ............................. 116 Table 17. Outputs from quality-of-life impact assessment required as input for other impact areas. .............. 120 Table 18. An overview of approaches and methods for evaluation of quality of life. ........................................ 121 Table 19. Pitfalls in evaluation of quality of life and best practices to avoid them. ........................................... 121 Table 20. Indicators recommended to be evaluated by every project addressing services and operation. ...... 125 Table 21. Outputs from services and operation impact assessment required as input for other impact areas. 126 Table 22. An overview of approaches and methods for evaluation of services and operation. ........................ 127 Table 23. Pitfalls in evaluation of services and operation and best practices to avoid them. ........................... 128 Table 24. Indicators recommended to be evaluated by every project addressing logistics. .............................. 132 Table 25. Outputs from logistics impact assessment required as input for other impact areas. ....................... 133 Table 26. An overview of approaches and methods for evaluation of logistics. ................................................ 134 Table 27. Pitfalls in evaluation of logistics and best practices to avoid them. ................................................... 134 Table 28. Indicators recommended to be evaluated by each project addressing transport activity and fleet composition. ....................................................................................................................................................... 137 Table 29. Outputs from transport activity and fleet composition impact assessment required as input for other impact areas. ...................................................................................................................................................... 138 Table 30. An overview of approaches and methods for transport activity and fleet composition impact assessment. ........................................................................................................................................................ 139 Table 31. Pitfalls in transport activity and fleet composition impact assessment and best practices to avoid them. .................................................................................................................................................................. 140 Table 32. Indicators recommended to be evaluated by each project addressing traffic safety. ....................... 144 Table 33. Outputs from traffic safety impact assessment required as input for other impact areas. ............... 146 Table 34. An overview of approaches and methods for evaluation of traffic safety. ........................................ 146 Table 35. An overview of approaches to simulation of safety-relevant scenarios. ............................................ 147 9 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table 36. Pitfalls in the evaluation of traffic safety and best practices to avoid them. ..................................... 148 Table 37. Indicators recommended to be evaluated by each project addressing traffic flow efficiency. .......... 151 Table 38. Outputs from traffic flow efficiency impact assessment required as input for other impact areas. .. 152 Table 39. An overview of approaches and methods for evaluation of traffic flow efficiency. ........................... 153 Table 40. Pitfalls in evaluation of traffic flow efficiency and best practices to avoid them. .............................. 155 Table 41. Indicators recommended to be evaluated by each project addressing energy and environment. .... 159 Table 42. Outputs from energy and environmental impact assessment required as input for other impact areas. ............................................................................................................................................................................ 160 Table 43. Overview of different methods that can be used for evaluation of energy and environmental impacts. ............................................................................................................................................................................ 161 Table 44. Pitfalls in the evaluation of energy and environment and best practices to avoid them. .................. 163 Table 45. Indicators recommended to be evaluated by every project addressing accessibility. ....................... 166 Table 46. Outputs from accessibility impact assessment required as input for other impact areas. ................ 168 Table 47. An overview of approaches and methods for accessibility impact assessment. ................................ 169 Table 48. Pitfalls in accessibility impact assessment, and best practices to avoid them. .................................. 170 Table 49. Indicators recommended to be evaluated by every project addressing land use impacts. ............... 174 Table 50. Outputs from land use impact assessment required as input for other impact areas. ...................... 175 Table 51. An overview of approaches and methods for land use impact assessment. ...................................... 176 Table 52. Pitfalls in the land use impact assessment and best practices to avoid them. ................................... 177 Table 53. Indicators recommended to be evaluated by each project addressing liveability impacts. ............... 180 Table 54. Outputs from liveability impact assessment required as input for other impact areas. .................... 182 Table 55. An overview of approaches and methods for liveability impact assessment. .................................... 183 Table 56. Pitfalls in evaluation of liveability and best practices to avoid them. ................................................. 184 Table 57. Indicators recommended to be evaluated by every project addressing impacts on economic activity and employment. ............................................................................................................................................... 187 Table 58. Outputs from economic activity and employment impact assessment required as input for other impact areas. ...................................................................................................................................................... 189 Table 59. An overview of approaches and methods for evaluation of economic activity and employment. .... 190 Table 60. Pitfalls in economic activity and employment impact assessment and best practices to avoid them. ............................................................................................................................................................................ 191 Table 61. Indicators recommended to be evaluated by each project addressing the socio-economic impact. 193 Table 62. An overview of approaches and methods for socio-economic impact assessment. .......................... 195 Table 63. Pitfalls in socio-economic impact assessment and best practices to avoid them. ............................. 196 Table 64. Factors of different dimensions of equity [114]. All of the factors in the table may vary across regions and change over time. ........................................................................................................................................ 201 Table 65. Examples of equity impact assessment for different indicators. ........................................................ 202 Table 66. An overview of approaches and methods for equity impact assessment. ......................................... 204 Table 67. Pitfalls in equity impact assessment and best practices to avoid them. ............................................ 206 Table 68. Template for reporting sustainability impacts in a sustainability table (a full template is available in the CCAM knowledge base). Colour coding is recommended for visualisation: green for positive impacts, red for negative, and grey for no impact. ................................................................................................................. 210 Table 69. Template for reporting sustainability impacts in a sustainability table across different scenarios (or CCAM systems, etc.), shown side by side. Colour coding is recommended for visualisation: green for positive impacts, red for negative, and grey for no impact. ............................................................................................ 210 16 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The FAME project supported stakeholder collaboration regarding knowledge sharing, consensus building, and facilitating data sharing in European CCAM projects. It built on previous projects: the EU-funded Coordination and Support Actions ARCADE 3 , CARTRE 4 , VRA 5 , FOT-Net 6 , and FESTA 7 . Common evaluation methodologies have previously been developed for driver support systems and driving automation. The most well-known is the FESTA methodology, developed for field operational tests (FOT) of advanced driver assistance systems (ADAS) and other vehicle information and communication technologies (ICT). The FESTA methodology was written for systems that can be used by ordinary drivers in their daily lives over long periods of time. It has been successfully used in large-scale FOTs in European and national projects. The handbook describing the methodology has been updated several times (most recently to version 8 in 2021 [2]) and remains a valuable resource when setting up and evaluating FOTs. With CCAM, a new approach for evaluation was needed, particularly where low technology readiness could not support such testing. Gaps have been identified in FESTA concerning its applicability for evaluation of CCAM. There are high expectations about the benefits of CCAM, including wide societal impacts, yet some concerns remain related to its use. Therefore, it is important to estimate the potential impacts of CCAM, even if FOTs are not yet feasible. EU-CEM was designed to evaluate CCAM systems at any stage of technological readiness. A few methodological frameworks specifically for CCAM evaluation existed before EU-CEM. The Trilateral Impact Assessment Sub-Group for Automation in Road Transportation [3] made the first effort to harmonise approaches and vocabulary in impact assessment globally. However, as a high-level framework, it does not cover all evaluation steps. Projects under the EU’s Horizon 2020 Framework Programme have built extensive evaluation methodologies, adapting the FESTA methodology for large-scale automated driving pilots (e.g. L3Pilot [5]). However, these methodologies were tailored for the specific project needs and are not generally applicable to all projects. These frameworks include several good practices. At the start of EU-CEM development, CCAM projects were interviewed to identify successful practices, shortcomings, and lessons learned, beyond what was documented in their reports. Based on the underlying reasons for the challenges faced, topics for guidelines and the project phases requiring methodological support were identified, forming the structure for EU-CEM. In addition, projectspecific best practices were generalised to be applicable to all projects. Thus, the insights and practices from the previous methodological frameworks and CCAM evaluations were adapted to establish the foundation for the common, generalised evaluation methodology of EU-CEM. 3 ARCADE (2018–2022). Aligning research & innovation for connected and automated driving in Europe (EU H2020 DT-ART2018, CSA, Project No. 824251). Horizon 2020. https://cordis.europa.eu/project/id/824251 4 CARTRE (2016–2018). Coordination of automated road transport deployment for Europe (EU H2020 ART06, CSA, Project No. 724086). Horizon 2020. https://cordis.europa.eu/project/id/724086 5 VRA (2013–2016). Support action for vehicle and road automation network (EU FP7-ICT-2013-10, CSA, Project No. 610737). Seventh Framework Programme. https://cordis.europa.eu/project/id/610737 6 FOT-Net (2008–2010). Field Operational Tests Networking and Interaction (EU FP7-ICT-2007-2, CSA, Project No. 224088). Seventh Framework Programme. https://cordis.europa.eu/project/id/224088 and follow-up Actions: FOT-Net 2 (2011– 2014). Field Operational Tests Networking and Methodology Promotion (EU FP7-ICT-2009-6, CSA, Project No. 269983). Seventh Framework Programme. https://cordis.europa.eu/project/id/269983 and FOT-Net Data (2014–2016). Field Operational Test Networking and Data Sharing Support (EU FP7-ICT-2013-10, CSA, Project No. 610453). Seventh Framework Programme. https://cordis.europa.eu/project/id/610453 7 FESTA (2007–2008). Field opErational teSts supporT Action (EU FP7-ICT-2007-1, CSA, Project No. 214853), Seventh Framework Programme. https://cordis.europa.eu/project/id/214853 17 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The first draft of EU-CEM was published for public consultation one year before its final version. Over 30 external experts provided feedback on the draft. To test the handbook with its target audience, a summer school was organised for junior members of CCAM evaluation teams. This feedback played a crucial role in finalising the handbook. 1.2. Scope of EU-CEM 1.2.1. Overall objective and scope The primary objective of the EU-CEM Handbook is to provide guidelines and share best practices for planning and conducting CCAM evaluation, especially impact assessment. Within the handbook, evaluation is defined as “the systematic process to research the amount, value, quality or consequences of an aspect related to a CCAM system and its use” and impact assessment as “the evaluation addressing the broader implications of the CCAM systems and their use on people and society”. The overall scope of EU-CEM is the evaluation and impact assessment of CCAM systems 8 . However, it is also beneficial for the development or deployment of CCAM when decisions between available alternatives are made based on their potential impacts. It is important to note that in such cases, clear objectives should be defined to allow ranking of the alternatives. This ranking is a political decision and falls outside the scope of EU-CEM. The following objectives were set for EU-CEM: • Ensure high quality evaluation of CCAM. • Provide common vocabulary and a common basis for CCAM evaluation in different projects. • Allow all CCAM projects to benefit from methodological lessons learned and best practices. • Support comparability and complementarity of evaluation activities. • Ensure exploitation of results of CCAM projects for future research and development. • Ensure high quality input for decisionand policymaking supporting both industry and authorities. EU-CEM puts particular emphasis on helping projects make robust and feasible evaluation plans in collaboration with those responsible for implementing the CCAM system in the project (referred to as the 'technical team' in this handbook). Thorough planning and effective collaboration set the foundation for successful evaluation and project outcomes overall. It is important to recognise that this handbook is not a substitute for expertise. Evaluation partners must possess the necessary (multidisciplinary) expertise in evaluation. This is also recommended for evaluation work package leaders. A common vocabulary is needed to ensure a shared understanding of the evaluation focus and objectives, the applied methodology and the results and conclusions drawn. To meet this objective, the EU-CEM Handbook is supplemented by the Taxonomy for CCAM [4], which contains the key terminology used in EU-CEM and CCAM in general. At the time of publication, the terminology of EU-CEM was aligned with the online taxonomy, which is regularly updated. For the latest definitions, it is recommended to refer to the key terminology from the Taxonomy tool provided in the Connected and Automated Driving Knowledge Base. Five high-level principles were developed for EU-CEM. They are related to setting the minimum requirements for evaluation (Guiding star principle), sharing best practices and lessons learned (Collaboration principle), using harmonised approaches (Comply-or-explain principle), encouraging agile ways of working (Agility principle), and being adaptable to project-specific needs (Flexibility principle). These principles are presented in Figure 2. 8 CCAM system may include a service concept or remote management. 18 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 2. Core principles of EU-CEM. 1.2.2. Type of projects and CCAM use cases within the scope of EU-CEM EU-CEM can be applied in three types of activities: (1) Ex-ante impact assessments conducted before the full deployment, where it can help to prepare for CCAM deployment and uptake or to identify unintended outcomes that may require mitigation (2) Ex-post evaluations to assess the impacts of already implemented CCAM systems (3) Design and deployment initiatives of CCAM systems with an aim to maximise societal benefits. EU-CEM is designed for large multi-partner projects but can also be applied to smaller ones. These projects may be funded by a customer, or they could be internal activities within companies or institutes. EU-CEM is most useful for projects where a technical team, separate from the evaluation team, is responsible for collecting (at least part of the) data in a field experiment, and especially if this is done at multiple sites. EU-CEM includes guidelines for project-internal communication, which is fundamental to creating a feasible evaluation plan that aligns with available resources. In smaller projects, differing objectives among partners are less problematic, simply because there are fewer partners and communication is easier. 19 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility EU-CEM is mainly intended for projects addressing CCAM systems with the following focus: • Automation in road transport of people and goods, covering all motor vehicle categories (specifically cars, trucks, shuttles, buses) and private, shared, and public transport • Higher-level driving automation (SAE [6] level 3 and higher), with and without connectivity • Use cases on public roads, automated driving systems with a specified operational design domain operating in mixed traffic. Additionally, EU-CEM is suitable for projects that include, for instance, testing of safety-critical scenarios on test tracks to supplement on-road testing, or driving in confined areas where the automated vehicle interacts with humans or non-automated vehicles. Projects focusing on technical testing or usability can utilise the basic guidelines for building a robust evaluation plan. Yet, it is good to acknowledge that EU-CEM addresses technical functioning and user aspects only from the perspective of providing necessary input for successful impact assessment. EU-CEM does not focus on the following use cases: assisted driving i.e. low-level driving automation (SAE levels 1–2), automation of rail, aircraft and watercraft transport, and Cooperative Intelligent Transport Systems (C-ITS) without driving automation. However, it is worth noting that most components of EU-CEM are not directly linked to a specific use case but can provide valuable input for CCAM evaluation cases beyond its core focus. 1.2.3. Evaluation areas covered CCAM can lead to different effects on the level of (1) single vehicles and (2) humans, on (3) the transport system and on (4) the society overall. This handbook covers all four levels. The EU-CEM Handbook provides evaluation area-specific guidelines for areas grouped under these four evaluation levels as described in Table 1. In this handbook, an evaluation area is defined as “a high-level category for topics addressed under evaluation activities”. These are technical evaluation, user evaluation and impact assessment. An impact area is defined as: “a field of study addressed under impact assessment activities”. Table 1. Evaluation and impact areas covered by the EU-CEM Handbook and their scope. Level of evaluation Area Scope Vehicle Technical functioning Software and hardware functionalities of CCAM systems, especially in terms of performance, cyber-security, and reliability. Driving behaviour Driving dynamics of a single vehicle, its longitudinal and lateral movements, and interactions with the environment and other road users. Human User Interaction of people (CCAM users and non-users) with CCAM system; their opinions, expectations and awareness of the technology, and how they use it. People mobility Individuals' potential and actual mobility choices; availability, access, quality, suitability, and affordability of mobility options. Quality of life Individuals' physical, mental, social, and financial well-being. 20 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Transport system Services and operation Business models and operation of vehicle fleets and transport services, effects on the operation of traffic management of the network. Logistics Efficiency, effectiveness, and adaptability of the logistics process, changes in freight transport, and material flows in supply chain processes. Transport activity and fleet composition Total amount of realised travel and transport; number and types of vehicles used. Traffic safety Number of accidents9 and injuries of different severities. Traffic flow efficiency Collective traffic patterns, travel times and throughput on a road network. Energy and environment Energy use and emissions of vehicles, air quality and noise pollution. Accessibility Potential of population groups to reach destinations and services at different times of day; the potential of businesses, facilities, and other activity centres to receive people and goods. Society Land use Spatial and functional implications, land utilisation patterns, including infrastructure development. Liveability Suitability or quality of living in the studied area(s). Economic activity and employment Economic development and labour dynamics, overall economic vitality, evolution of businesses and the job market, and skill requirements of the workforce. Socio-economics Economic and societal outcomes of CCAM, benefits and costs, societal and economic value. Equity Fairness, impartiality, and inclusivity, how impacts are distributed across different population groups, regions or generations. Sustainability Environmental integrity, social well-being, and economic vitality, capacity to meet current needs while not compromising resource and opportunity availability for future generations. 9 During the writing of this handbook, it was noted that the terms ‘a crash’ or ‘a collision’ were often used interchangeably with ‘an accident’. It is perfectly acceptable to use any of these terms. 21 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 1.3. Structure of the EU-CEM Handbook The EU-CEM Handbook provides guidelines for three different phases of a project: 1. When writing the project proposal: Chapter 2 provides guidelines for laying the ground for successful evaluation in the proposal preparation phase. It includes guidelines written for the partners responsible for the evaluation in the project (referred to as 'the evaluation team' in this handbook), coordinators, and proposal evaluators. 2. When the project starts: Chapter 3 provides guidelines for setting an evaluation plan that is feasible to execute, regardless of the evaluation areas in focus (technical evaluation, user evaluation, impact assessment). Chapter 4 describes evaluation-area-specific guidelines to support this planning. Chapters 3 and 4 are intended for the evaluation team. 3. When the project ends: Chapter 5 provides guidelines for reporting evaluation outcomes, and for providing lessons learned, new best practices and feedback for future EU-CEM updates. Chapter 5 is intended for the evaluation team. The role of each chapter and their outputs are described in Figure 3 below. The outputs include the project plan (Description of Action), an agreement on what data to collect and how, the evaluation plan, and feedback for future EU-CEM updates. Figure 3. Role of different chapters of the EU-CEM Handbook (light green) and their outputs (dark green). 22 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 1.4. Compliance with EU-CEM EU-CEM is intended for the evaluation of CCAM use cases in road transport with higher-level driving automation. One of the principles that guided the development work was flexibility. To be applicable for different CCAM evaluations, EU-CEM must leave room for projects to adapt it to their specific scope and needs. This raises the question of how to determine whether a project is compliant with EU-CEM. To claim compliance, the project must meet these minimum requirements: • The project includes evaluation activities, specifically impact assessment, as defined in the EU-CEM Handbook (see the first paragraph of Chapter 1.2.1). • The evaluation plan is prepared by the evaluation team following the guidance in Chapter 3, alongside the planning of all other project activities, at the beginning of the project. High-level planning is already conducted during the project proposal preparation phase, in accordance with Chapter 2. Specifically, the evaluation planning is not left until after the planning of potential field experiments, nor is it carried out by anyone outside the evaluation team. • The guidelines of the EU-CEM Handbook are followed. If a specific guideline cannot be followed, an explanation for the deviation is given (comply-or-explain principle). • A line of communication between the evaluation team and other project partners is established (and utilised during the project’s lifetime). • The results are analysed from a sustainability perspective in accordance with Chapter 4.4.6. • The project utilises the Taxonomy for CCAM. • The project contributes input and feedback for future updates of the EU-CEM Handbook and provides recommendations for future research needs in the evaluation report. 1.5. Outlook The first complete version of the EU-CEM Handbook was published in 2025. The methodology is intended to evolve over time, incorporating new learnings and experiences. The next update is expected in 2028. 23 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 2. Guidelines for the project preparation phase This chapter provides guidelines for preparing the evaluation in a CCAM project, focusing primarily on the proposal’s preparation phase, but also on the project’s initial stages. These guidelines help establish a strong foundation for successful evaluation. They specifically address topics that have proven challenging in previous CCAM projects. General project management issues that are not specific to CCAM evaluation are not covered, as information on these topics is available in many other publications. In addition to the evaluation team, the guidelines below are also useful for coordinators and project proposal evaluators. 2.1. Scoping of evaluation The expected impacts of CCAM are wide-ranging. As it is not possible to assess all potential impacts, scoping and prioritisation are necessary. Finding agreement already in the proposal phase ensures that project partners share a common understanding on how to address project goals. Scoping ensures that the commitments made during the proposal phase can be executed. That is, that the consortium’s expertise and tools align with the project’s scope and the scope aligns with the consortium’s expertise and tools. Scoping is first required for the project as a whole and then specifically for the evaluation, ensuring consistency with the overall scope. The project proposal must clearly demonstrate how evaluation activities align with the overall project scope. The guidelines below cover the following topics: • How to set the overall scope of the evaluation • How to choose impact areas for evaluation • How to build a consortium fit for the evaluation scope Guideline 2.1. How to set the overall scope of the evaluation The overall evaluation scope should align with the overall project goal and the overall project scope. Scoping of evaluation requires understanding the overall project goal and scope, the state of the art of CCAM, its potential benefits and adverse effects, and being realistic about what can be tested within the project. Both the overall project goal and scope are derived from the call text by extracting the essence of the call, from the objectives that the client has for it. These objectives may include obtaining information for policy and investment decisions, facilitating and stimulating the learning process and knowledge exchange, and paving the way towards deployment of CCAM. Note that in this handbook the term “call text” refers to any document outlining the expectations for the actions to be carried out in the project. This may include an EU research and innovation programme call text, any request for a proposal, or any other expressed wish of the client. Define the key terminology in the proposal. Misunderstandings related to the interpretation of the call text or the scope of the project and evaluation are hard to resolve afterwards. For clarity, define what is excluded from the scope to leave less room for ambiguity, even if these exclusions are not included in the proposal. The scoping of evaluation starts by specifying the targeted level of evaluation in the proposal: will it focus on the level of single vehicles and humans, the transport system, or society? The scope may extend beyond realworld testing capabilities. For example, if the project conducts field tests of CCAM vehicles in limited use cases, the scope of evaluation can still include broader impacts, such as regional effects. The evaluation scope can be structured around numeric targets, expected key impacts, or societal challenges, depending on the call text. When the call text allows flexibility in scoping, stakeholder analysis or 24 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility consultation can help refine it. Ensuring alignment with a clear demand for new knowledge will maximise the project’s impact and relevance. The evaluation scope can also build upon earlier evaluations by addressing identified knowledge gaps, unresolved challenges of earlier projects, or expanding previous evaluations to new impact areas, scenarios or stakeholder groups. Additionally, the scope may include repeating certain evaluations to improve the validity and reliability of important impact estimates. This can be achieved with systems of higher technological readiness, larger or better-quality datasets (e.g. improved evidence from the field), enhanced tools or increased resources for evaluation. In some cases, call texts may set unrealistic expectations regarding the impacts of CCAM. When this occurs, a balance must be found between responding to the call and making realistic commitments. Example Call text: “The overall objective of this call is to promote a wide market introduction of highly automated driving systems towards SAE level 4.” [7] The consortium sets the following overall goal for the project: “Make driving automation robust and reliable by taking intelligent vehicles technology to conditions and scenarios neither extensively tested nor demonstrated earlier in European and overseas traffic.” [8] In practice, the high frequency of take-over requests was identified as one of the main challenges in achieving this overall objective of the call. Consequently, the consortium decided to aim for “defragmentation and extension of the operational design domains (ODDs)” [9] by introducing technology enablers to support automated driving. The overall scope of impact assessment was set to assess “the impacts of highly automated driving […] and what is the contribution of the technology enablers on the impacts” [10]. Guideline 2.2. How to choose impact areas for evaluation The details of the evaluation will be defined in the evaluation plan (Chapter 3). However, it is important to determine already during the project preparation phase which impact areas (overview in Table 1, full details in Chapter 4) will be covered and the main approaches for their assessment. This early decision-making ensures sufficient resources are allocated for evaluation—especially for expensive or time-consuming approaches like field experiments, driving simulators, or extensive traffic simulations— and enables the formation of a consortium suitable to the task (see Guideline 2.3). Once the consortium and budget are fixed, flexibility in choosing the impact areas becomes more limited. Balancing what impact areas to cover and what resources to allocate to evaluation with other project activities is an iterative process. Prioritising impact areas based on their importance in addressing the overall scope of evaluation and the potential for impact allows researchers to allocate their resources and efforts more effectively. This helps to ensure the evaluation is efficient and the findings are relevant and impactful. To choose which impact areas to cover, check the call text to see if it specifies impact areas directly or lists impacts or indicators that must be assessed. Then, assess whether the high-level scope set for evaluation determines the impact areas (see Guideline 2.1). Both top-down and bottom-up approaches should be used when determining relevant impact areas: Top-down: If the call text explicitly sets requirements for impacts that should be addressed, the corresponding impact areas should be included in the evaluation. This often means incorporating other impact areas from which input is required for the evaluation of those areas set by the call. For example, traffic safety impact assessment requires input from driving behaviour evaluation. If addressed, start with the 25 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility society-level impact areas, working towards the impact areas included in the transport system, human and vehicle levels. Bottom-up: If the call text does not specify impact areas, the selection process can start from the CCAM system itself by considering how it directly affects road users and driving behaviour, and how these changes might affect traffic and transport and, ultimately, how they impact the whole society. Drafting impact pathways, from aspects directly affected by CCAM to the impact areas in the evaluation scope, can help in this process by identifying linkages and dependencies between impact areas and levels of evaluation. Use theories and literature to establish these connections. The illustrations in Chapter 4 may be used as supporting material. Prioritise the areas where substantial impacts, benefits and burdens are expected. These pathways are a good tool to check against unrealistic expectations and ensure that the chosen evaluation scope produces outputs that can be delivered. It is good to acknowledge that the impacts of CCAM can be so far-reaching that no single project can address them all. Therefore, it is advisable to focus on fewer topics and evaluate them thoroughly. The availability of resources for evaluation should align with the choices made and the estimated resources needed. The impact areas described in this handbook (see Chapter 4) relate to one or more pillars of sustainability: economy, society, and ecology—or people, planet, profit. Understanding how CCAM systems affect sustainability is important. Regardless of the chosen impact areas, all projects are strongly encouraged to reflect on their results from a sustainability perspective. Guidelines for this are provided in Chapter 4.4.6. Example Top-down approach: The call text mentions the following topics that the project should address: Road safety, energy use, pollutant emissions, traffic congestion, equity. The corresponding impact areas are included in the evaluation. Next, the pathways to the listed impact areas are drawn. For example, to see how the impacts are distributed among different population groups (impact area: equity), it is important to know the most potential users and non-users, in other words, acceptance and possibilities for CCAM use (user) for different groups. Therefore, all these aspects should be studied separately. Bottom-up approach: CCAM influences comfort of travel and value of travel time (user). This leads to changes in travel patterns (people mobility), and consequently in vehicle kilometres travelled (VKT, transport activity and fleet composition). The impact on total VKT influences the total impact of CCAM on the energy and environment at transport system level. Scaling up effects at single vehicle level (emissions per distance driven) to traffic flow impacts usually requires simulations, which need information on changes in driving behaviour for proper model calibration. Finally, the impacts of CCAM on sustainability will be assessed. The sustainability perspective provides a way to draw overall conclusions of the impact across all results. In summary, the impact areas chosen in this example, using EU-CEM terminology, are: Driving behaviour, user, people mobility, traffic safety, energy and environment, traffic flow efficiency, equity, sustainability. 32 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility • Communicate effectively with partners to align expectations in light of feasibility, timeline, and resource constraints. • Address potential misunderstandings about the evaluation process early on. Example To ensure continuous alignment, a structured information exchange between work packages is established. Biweekly meetings are scheduled from the start of the evaluation planning phase until all data has been analysed. These meetings facilitate: • Dialogue between the evaluation team and related activities, such as technical and test site teams, as well as within the teams. • A shared understanding of the overall project, enabling stakeholders to respond to plans and updates effectively. • Regular updates on the progress of data collection or analysis. • Discussions on any technical issues affecting evaluation. • Alignment of expectations, monitoring the progress of milestones, identification of risks. Guideline 2.8. How to enable adaptivity of the evaluations CCAM and related regulations evolve rapidly, sometimes necessitating adjustments to CCAM test planning and execution during the project. As such changes may affect the evaluation, the evaluation scope and structure must be designed with adaptivity in mind (see the example for Guideline 2.1). In practice, the highlevel project plan outlined in the proposal will be elaborated into an evaluation plan early in the project. Revisions may be required during the project due to changing needs, possibilities, and challenges. Fundamental adaptations require approval from the consortium leader and the client. Start evaluation planning early in the project and adapt to what is feasible at test sites (see Guideline 2.5 for milestones and timeline). Feasibility may change, so work in close collaboration with test sites until the field experiments are completed. In practice, this will be an iterative process (see Chapter 3 introduction). When preparing the project proposal, it is important to recognise that iterations will be required during the project’s lifetime before the evaluation plan is finalised. To ensure that the evaluation team has sufficient time to complete their tasks, a cut-off date should be established, after which major adaptations can no longer be accommodated. The cut-off date should be set based on experience regarding the time required for the planned key approaches and methods, as outlined in Milestone 5 in Guideline 2.5. Example An example of ensuring adaptivity to changes in test site plans is to extend the evaluation methodology task until the planning phase of the field experiments is completed, or until successful data collection can be reasonably verified. Guideline 2.9. How to ensure the quality of the evaluation Each project strives to deliver high-quality results. Several project governance mechanisms help ensure the quality of evaluation activities, and these should be discussed during the project proposal phase and included in the work plan: • Monitoring the quality and validity of the evaluation plan by members of the evaluation team with the most experience of CCAM evaluations. The evaluation manager or evaluation work package lead 33 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility should assign these responsibilities. The plan should be communicated to all relevant stakeholders in the consortium. Identify and communicate key decisions in the evaluation plan (e.g. assumptions made in the evaluation). Early communication with project partners ensures that everyone is aware of critical decisions and has the opportunity to seek clarifications or suggest changes. Failure in this regard can lead to inconsistencies that may only become evident after the results have been finalised. • Testing all steps of the evaluation process beforehand should be facilitated by the project plan, ensuring that initial data collection and analysis can begin as early as possible. Clearly describe the plans for this in the project proposal. • ‘Evaluation partnerships’ (see the example of Guideline 3.18) can be used for discussing interim results and identifying missing, unlikely or unexplainable results, especially when analysing realworld data. Allow sufficient time for these collaborative reviews to check for potential inconsistencies and to interpret the results accurately. These partnerships can also assist in planning the reporting of results, defining both the desired content and the presentation format. Example In the proposal for a CCAM project, the evaluation team ensures the quality of the evaluation as follows: • Experienced experts are involved in the evaluation work packages both for development and execution of the evaluation plan. The proposal explicitly requires that an experienced expert review the evaluation plan early in the project, with necessary revisions made based on their feedback. • The proposal includes a phase for testing the evaluation process using preliminary data from the test sites. Insights from this will be used to refine the evaluation plan and tools before the full-scale evaluation begins. • The proposal explicitly requires interim result reviews to discuss the preliminary findings. These reviews are both internal and external to the work package. • The proposal allocates sufficient time at the end of the project for comprehensive work-packageinternal and -external reviews of the reports. An internal milestone is set, requiring the reports to be finalised at least one month before the submission deadline to allow for these reviews. Guideline 2.10. How to set up a process for mediating conflicts CCAM evaluations typically involve multiple project partners working together towards shared goals. In such large, multi-partner or complex projects, conflicts may arise despite robust information exchange processes. These conflicts often stem from differences in expectations, goals, values, perceptions, and knowledge levels among team members and organisations. To effectively manage such challenges, a conflict resolution mechanism should be incorporated into the project preparation phase. A well-defined dispute resolution mechanism improves the ability to manage potential conflicts without compromising project objectives or causing significant delays. Common dispute resolution mechanisms include, for instance: • Negotiation: Direct discussion between conflicting parties to find a mutually agreeable solution. • Mediation or conciliation: A neutral third party facilitates discussions between conflicting parties to reach a voluntary resolution. • Arbitration (non-binding or binding): A third party reviews the dispute and provides a decision. Nonbinding arbitration offers a recommendation, while binding arbitration is a final decision the parties must accept. Conflict resolution mechanisms are needed for situations where conflicts cannot be resolved within or by the evaluation team. This involves designating a ‘conflict resolver’, who may be the evaluation work package 34 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility leader, the project coordinator or a specifically appointed person, to liaise with the project coordinator and work package leaders. Unclear communication lines often contribute to conflicts. Defining and actively using structured communication channels minimises this risk. This is why EU-CEM specifies necessary lines of communication for the planning of evaluations (see Chapter 3). In addition, ensure that all project partners are aware of their designated points of contact for conflict resolution and the steps involved in escalating unresolved issues. Example 1. Conflict within the evaluation team: An evaluation partner does not adhere to the agreed societal scenarios, making it impossible for other evaluation partners to use their results as input. In these work package internal issues, the conflict resolver is the evaluation work package leader. A practical way to address this issue is by organising workshops to define the societal scenarios, allowing partners to indicate essential (must-have) and desirable (nice-to-have) features from the perspective of their activities. This collaborative process helps build consensus on setting the societal scenario or establishing several alternative ones. Once consensus is reached, all evaluation areas can proceed with detailed evaluation planning, ensuring alignment across the project. 2. Conflict between different work packages: The evaluation team requests specific data from the test site team. Initially, the data providers agree to deliver the requested data. However, later in the project, they report that providing the data is not possible due to technical challenges. For the evaluation team, the lack of necessary data undermines the validity and completeness of the evaluation outcomes. For the test site or technical teams, challenges related to data collection may exceed the available resources or may have arisen from unexpected technical limitations. As multiple work packages are affected, a project level conflict resolver, such as the project coordinator, should facilitate finding a solution. For effective mediation, private discussions between the conflict parties and the resolver are preferred to more public disputes or confrontational messaging. Close and continuous dialogue between the evaluation team and data providers can help prevent such conflicts. 35 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 3. Guidelines for developing an evaluation plan This chapter provides guidelines for developing an evaluation plan in CCAM projects. When mutual understanding and agreement among the project team has been reached on the characteristics of the CCAM system to be evaluated, the iterative process for developing the plan begins. If new information on the CCAM system is acquired, the evaluation plan must be adjusted accordingly, leading to iterations. The main elements of the evaluation plan are: • Description of the CCAM system under evaluation (Chapter 3.1) • Research questions (Chapter 3.2) • Evaluation methods (Chapter 3.3) • Data specification and tools (Chapter 3.4) • Experimental design (Chapter 3.5). These elements are interconnected, making the evaluation inherently iterative. Adaptations are often needed for each of these elements based on feedback or requirements identified during the development of the others. This process continues until the evaluation plan is finalised. Figure 4 below describes these iterations. Guideline 3.28 in Chapter 3.6 describes how to compile the evaluation plan once a feasible solution has been found for each step. 36 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 4. Workflow and iterations in setting the evaluation plan until a feasible plan is established. 3.1. CCAM system description The first step in developing an evaluation plan is to clearly describe the CCAM system that will be addressed in the evaluation, along with any related service concept where relevant. In projects addressing multiple CCAM systems or service concepts, each must be described individually. This chapter provides a common system description for all evaluation areas. It is important to recognise that assessing the wider socio-economic impacts of CCAM tested in field experiments may not be meaningful if these experiments involve low technology readiness level (TRL) systems. Instead, evaluation, especially impact assessment, may focus on a higher-TRL version of the system when implemented at scale. 37 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Therefore, two descriptions may be needed: one for the CCAM system tested in field experiments and another for a higher-TRL version of the system when CCAM is fully deployed. It is important to understand and clearly document the differences between these two descriptions. Due to the diversity of CCAM systems, each project must determine the relevant aspects to describe, using the guidelines below as a starting point. These system descriptions guide the development of research questions, the establishment of experimental design, and the planning of data collection. They also provide context when forming conclusions from evaluations and help external stakeholders understand and interpret the study and its results. The guidelines below cover the following topics: • How to describe the CCAM system • How to describe the ODD of an automated driving system • How to describe the CCAM service concept • How to describe the gap between the CCAM system tested in field experiments and the system used for the impact assessment These steps may be linked and require iterations; they are not hierarchical steps done in sequence. Note that the best chronological order of tasks may differ from the sequence of guidelines below, depending on the project. Guideline 3.1. How to describe the CCAM system Describe the CCAM system in terms of the following elements, with additional elements added when relevant to the specific project. The descriptions should be short but precise. • Vehicle category: passenger car, shuttle or minibus, bus, truck, etc. • Level of automation (SAE level [6]): If the system does not align perfectly with any SAE level, describe the closest match and explain any discrepancies. • Automated driving system type: e.g. highway chauffeur • Connectivity: e.g. communication technology, communicated information • Capabilities of the automated driving system, covering for example: • Dynamic driving tasks that the vehicle is able to perform and is not able to perform in automated mode (e.g. lateral and longitudinal vehicle control, monitoring the driving environment, object and event response execution) • Type of sensors installed, sensor ranges and operating angles • Target speed range • Target time headway, possible time headway settings • Human-machine interface (HMI), for example related to take-over requests—when and how they are directed to the driver; and activation and deactivation of the automated driving system—how the mode status is indicated, and what is required from the driver) • Minimal risk manoeuvres, such as coming to a standstill on the side of the road • Technology Readiness Level (TRL 1-9): In case of low TRL, whether safety drivers or other necessary safety procedures (e.g. remote monitoring) is required specifically due to low TRL • Traffic management and operation required for the CCAM use case, including physical, digital, and operational infrastructure 38 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Example An automated driving system for passenger cars (adapted from [11]): Motorway Automated Driving System: Vehicle category Passenger car Level of automation SAE level 3 Automated driving system (ADS) type Highway chauffeur. ADS is capable of driving on a motorway in the current lane. Connectivity Vehicle-to-vehicle communication (ETSI MCS) Capabilities of the automated driving system Dynamic driving tasks In automated mode, the system is able to keep the vehicle within its lane and maintain a safe distance to vehicles ahead while in automated mode. It is not able to perform overtaking manoeuvres in automated mode. Sensors, ranges and operating angles Midand long-range radar, shortand long-range camera, LiDAR Target speed range Speed limit up to 120 km/h Target time headway 1.6 s HMI: Activation by user The driver activates the ADS by pressing a button on the steering wheel. Signature auditory sound that automated driving mode is activated. HMI: Deactivatio n by user The driver deactivates the ADS by pressing a cancel button on the steering wheel or through steering input or pedal operation. Signature auditory sound that automated driving mode is deactivated. HMI: Automated driving mode status Automated driving mode status is shown in the instrument cluster, on the steering wheel and on the centre display. 39 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Takeover request If the system needs to deactivate automated driving mode, the ADS issues a take-over request to the driver. The request shows in the instrument cluster, on the steering wheel and on the centre display together with signature auditory sound indicating that the driver should take over. The belt tensioner tugs on the driver's seatbelt if the driver does not respond to a take-over request. Minimal risk manoeuvres Safe stop, safety driver to take over control. Technology Readiness Level Low TRL. The driver must be a trained safety driver. Traffic managemen t and operation No dedicated traffic management required. Guideline 3.2. How to describe the ODD of an automated driving system The operational design domain (ODD) describes the “operating conditions under which a given driving automation system is specifically designed to function. This includes, but is not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.” [12] Specify the conditions under which the ADS can and cannot operate, considering for example: ▪ Road type: type of public road, closed areas or dedicated corridors ▪ Traffic rules and control: speed limit, traffic signs, traffic lights, overtaking allowed ▪ Infrastructure elements: intersection types, railroad crossings, separation of driving directions/lanes, slope of road, tunnels, bridges, presence of roadside parking or roadworks, connectivity, availability of space for minimum risk manoeuvres ▪ Quality of infrastructure: lane markings (e.g. faded), traffic sign readability, road surface (potholes, longitudinal ruts), quality of digital infrastructure ▪ Weather and road conditions: precipitation, temperature, road surface condition (e.g. standing water, snow, ice), fog ▪ Traffic conditions: traffic volume, presence of pedestrians and cyclists in lane ▪ Lighting conditions: daylight, darkness, road lighting Describe other relevant aspects as needed, such as abilities for border crossing. Example Example of the ODD specified for an urban automated driving system (ADS) for passenger cars (adapted from [13]): The ADS operates on main public streets with separated driving directions and speed limits up to 50 km/h. It is capable of navigating signalised intersections, but non-signalised intersections and all rail-road crossings are outside its ODD. Roadworks, temporary lane changes, and construction zones are also outside its ODD. The ADS requires visible lane markings or a curb on at least one side. 40 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The ADS operates in good weather conditions and in light or normal rain (up to 7.5 mm/h). Heavy rain (>7.5 mm/h), snow, fog, and extreme weather conditions, as well as icy or snowy road surfaces, are outside the ODD. The ADS operates in all traffic conditions. It works in daylight and at night, with and without road lighting. Further reading ISO (2023). ISO 34503:2023 Road Vehicles—Test scenarios for automated driving systems— Specification for operational design domain. https://www.iso.org/standard/78952.html SAE International (n.d.). J3259 (WIP) Taxonomy & Definitions for Operational Design Domain (ODD) for Driving Automation Systems. (Integrated in the ISO standard above) Koopman, P., & Fratrik, F. (2019). How Many Operational Design Domains, Objects, and Events? Safe AI 2019: AAAI Workshop on Artificial Intelligence Safety, Jan 27, 2019. https://ceur-ws.org/Vol-2301/paper_6.pdf Guideline 3.3. How to describe the CCAM service concept The description of the CCAM system under evaluation should include a description of the service concept in terms of the following, where relevant: • Type of service: private transport, public transport, taxi, on-demand transport with fixed (flexible) schedule and fixed (flexible) route, freight transport services • Vehicle ownership model: vehicles owned privately, by a company or by a public entity • Vehicle use model: private use, shared uses or shared rides • Area of operation, size: city centre, built-up area, rural-area, inter-regional operation, size in terms of area (km2) or length of routes (km) • Fleet operation and management: remote operation, remote supervision, vehicle routing • Service description from a user’s perspective: what are the requirements for use and the steps in using or travelling • Business model, including costs for the user: as part of local public transport, subscription based, payment per ride/km • Target population: All, older adults, disabled people, schoolchildren, etc. Describe other relevant aspects as needed. Example A service description for an automated shuttle service providing last-mile connection to and from a metro station: Type of service Public transport, fixed route, fixed schedule Ownership model Owned by the regional public transport operator Use model Shared rides Area of operation Lauttasaari area in Helsinki, 4 km2 Fleet operation and management Remote supervision of the fleet, centralised routing 41 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Description of service from the user's perspective The user can board the shuttle at one of the predefined stops along a predefined route through a residential area, leading to a metro station. A shuttle departs from the tram stop every 10 minutes. No driver or operator is present on the shuttle, but it is connected to a remote operation centre that users can contact via a screen in the shuttle. Business model The public transport operator receives a normal public transport fee from the users; the service receives the same subsidies as the local public transport Guideline 3.4. How to describe the gap between the CCAM system tested in field experiments and the system used for the impact assessment Field experiments in CCAM projects typically involve low technology readiness level (TRL) systems that require additional safety measures, such as safety drivers. However, impact assessment focuses on high-TRL systems that are widely adopted and used by the general public as drivers or passengers. Therefore, the fully developed CCAM system envisioned in impact assessment needs to be defined separately from the one tested in field experiments, using the instructions of the previous guidelines. Here, their main differences and the resulting implications of these differences must be specified. This should consider: • Capabilities of the CCAM system, for example in terms of target speed and speed limit, headway, human-machine interface (HMI), and dynamic driving tasks. • Operational design domain and operating domain, compare the conditions under which the CCAM system is designed to operate, is tested, and is assumed to function for impact assessment. • Safety procedures, for example how fallback procedures differ. • Infrastructure support or requirements, covering physical, digital, and operational road infrastructure, traffic management. • Service concept, such as being part of public transport, on-demand transport, robotaxi, shared vehicles for private use. Clarify the implications for impact assessment by explaining how these differences affect the evaluation. This should allow determining: • Which field experiment data can be used in impact assessment and for which research topics. • Where adjustments are necessary to account for the differences. • Which parts require additional modelling or assumptions due to the gap. Example This example illustrates how to identify and describe the gap between a tested system (low-TRL automated passenger car) and the assumed system (high-TRL automation for impact assessment). Aspect Tested system Assumed system in impact assessment Differences and implications Dynamic driving tasks Safety driver performs lane changes manually, other dynamic All dynamic driving tasks are automated. Gap: The characteristics and frequency of lane changes in the tests does not represent true automated behaviour. Implications: Data related to lane-change driving behaviour cannot be used directly in 48 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4. Feasibility: Prioritise questions that are realistically answerable within the project’s available resources, tools, timeline, experimental design, and data (see Guideline 3.8). Consider the complexity and development needs of the required research methods. Example Consider a CCAM project focused on the impacts of automated driving on urban mobility. Initially, the consortium proposed 15 research questions. However, due to limited resources, it uncertain whether all of them could be properly evaluated. Thus, the evaluation team followed the four steps outlined in the guideline to prioritise the list and exclude research questions of low priority: Step 1: Relevance. • The team brainstormed the research question for all impact areas within the evaluation scope (see the example in Guideline 2.2). • Research questions that did not directly align with the project's goal of assessing impact on urban mobility were deprioritised. • Example: Questions focused on rural areas were deprioritised, as the project specifically targeted urban environments. Step 2: Importance. • Through a literature review and experience with past projects, the evaluation team identified the most likely impact mechanisms related to CCAM on urban mobility. • Example: Research indicated that automation in privately-owned cars could increase car travel, affecting congestion. Thus, questions such as “How does CCAM affect urban congestion patterns?” were prioritised. Step 3: Dependency within the hierarchy. • The team mapped interdependencies between research questions based on the impact pathways (see Chapter 4). • Example: Questions on driving behaviour and interactions between different vehicle categories in traffic were seen as necessary for evaluating traffic flow efficiency. • Thus, foundational questions like “How does CCAM influence car-following behaviour in urban settings?” were prioritised, as they supported answering the most relevant and important research questions. Step 4: Feasibility. • The evaluation team assessed available resources, tools, and data collection and analysis methods. Each question’s resource requirements were cross-checked against them. • Example: Some questions required resource-intensive tool development or extensive new data collection, which exceeded the project’s budget and timeline. • Questions that could not be answered within the project’s constraints were excluded from the final list. After applying these prioritisation steps, the evaluation team had identified eight research questions that were relevant, important, and feasible given the resources. The rest were documented but not pursued within the project’s evaluation plan. 49 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Guideline 3.8. How to check the feasibility of research questions Once a draft evaluation plan is ready, it is essential to check the feasibility of each research question. Feasibility checking ensures that the evaluation can be executed effectively within the project’s resources, timeline, and technological constraints. If changes to the project plans occur at later stage, such as during the testing phase, the feasibility of the planned research questions should be re-evaluated. This process involves assessing several critical components: 1) Data provision (see Chapter 3.4): Verify that all required data for evaluation are covered by the data-sharing agreements. Ensure that the planned external data sources are available, accessible, and appropriate, and that the provided quantity of data is sufficient for the methods foreseen. Confirm the required inputs for relevant methods and tools, their calibration and validation, and proper contextual data. 2) Experimental design (see Chapter 3.5): Ensure that the experimental design meets the minimum requirements of the envisioned methods, including the type and number of participants, as agreed with the test site teams. Check if the technological readiness level of the tested system is sufficient for the envisioned testing. 3) Evaluation tools and methods (see Chapter 3.3): Confirm that the necessary tools are available to the responsible evaluation team members. If tool development is required, verify that the planned development timeline and resource allocation (time, funding, personnel) are reasonable. 4) Timeline alignment (see Chapter 3.6): Verify that the time allocated for each element of the evaluation process is realistic. Consider the dependencies between different stages of the evaluation and how they align with the overall proposed project timeline. 5) Resource availability (see Chapter 3.6): Check that evaluation team partners are available for evaluating each research question with sufficient resources (time, researchers, budget). If important research questions are deemed unfeasible, check the following adaptations: • Is it possible to adapt the data collection (see Chapter 3.4)? For example, by acquiring additional data sources or modifying the existing data collection process to meet the requirements. • Is it possible to adapt the experimental design (see Chapter 3.5)? • Can the team develop new methods or tools or adapt the existing ones (see Chapter 3.3)? • Can the research question be reformulated (see Guideline 3.6)? If no feasible solution can be found, the research question should be removed from the evaluation plan. To build a resilient evaluation plan, prioritise high-level research questions that: • Have a feasible plan for execution. • Do not depend on a single researcher, data provider, or unique tool. • Require minimal development of new evaluation tools. Higher risks and more uncertainties can be tolerated for low-level RQs unless they are vital to answering a higher-level RQ. Example These examples illustrate how a CCAM project conducted feasibility checks from the five perspectives outlined in the guideline, along with examples of solutions to overcome different barriers to evaluation. 50 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Data provision: • Challenge: A research question focused on the CCAM system’s operational design domain (ODD) in varying weather conditions. However, no weather or road condition data were available from the test sites, preventing the evaluation of this research question. • Solution: An external source for road and weather condition data was identified, allowing the research question to remain in the evaluation. Experimental design: • Challenge: A research question addressed user acceptance of the CCAM system with the intention to study users’ opinions after experiencing the system in a field experiment. However, only professional safety drivers employed by the developer of the system were allowed to operate the test vehicles. Their experience does not reflect that of ordinary users. • Solution: The experimental approach was adjusted to place the participant in the passenger seat instead of the driver seat. This change retained the research question, and limitations from experiencing the system as a passenger rather than as a driver were planned to be addressed in the evaluation report. Evaluation tools: • Challenge: A research question focused on traffic safety required simulation-based evaluation. However, one partner lacked validated safety simulation software. • Solution: The partner committed to investing resources (time, funds) into validating their in-house simulation tool. Two backup plans were developed in case the validation failed: it was ensured that another partner also had the means to conduct these simulations and that an alternative tool could be used for the evaluation. Timeline: • Challenge: A research question required the results of other research questions for its evaluation to even begin, and also several months of execution time. This would exceed the total time allocated for evaluation in the project, meaning this research question was deemed unfeasible. • Solution: The timeline was adjusted by scheduling the assessment of the other research questions a few months earlier. Resources: • Challenge: A research question required access to a driving simulator, which was costly and not included or budgeted in the original project plan. • Solution: The project coordinator reallocated the budget through an amendment to the project agreement, allowing the research question to remain in the evaluation plan. Guideline 3.9. How to set hypotheses Hypotheses are statements about the state of the world and about the causal relationships between variables. Scientific hypotheses should be both falsifiable and testable. Falsifiability means that the hypothesis can be proven false, while testability means that the hypothesis can be evaluated based on collected data. A hypothesis supported by evidence is typically accepted as true until new evidence potentially refutes it. Hypotheses should describe what will be measured and estimate the magnitude of the impact. The measure can be either quantitative (e.g. percentage change in travel time) or qualitative (e.g. increase, decrease or no effect). The expected magnitude might involve precise values, like “a 20% reduction in travel time”, or general trends, such as “an increase in user satisfaction.” When applicable, hypotheses should describe the conditions 51 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility under which the impact is expected and the causal mechanisms that explain why the anticipated impact is expected. Hypotheses can be derived from previous research or propose entirely novel ideas. In both cases, the link between existing knowledge and the current hypothesis should be explained to position the evaluation within the broader scientific context. Deriving hypotheses can contribute to the development and phrasing of research questions. A single research question may give rise to several hypotheses. Well-articulated hypotheses help design experiments (Chapter 3.5), structure data collection (Chapter 3.4) and analysis methods (Chapter 3.3) to answer research questions, though they are not mandatory for CCAM evaluation. They are valuable when the goal is to test specific causal relationships or quantify how independent variables (e.g. CCAM implementation) affect dependent variables (e.g. travel time, number of accidents). However, hypotheses may not be suitable for purely exploratory questions or for research focused on describing properties, phenomena, or simply measuring values without establishing causal links. Ultimately, whether to use hypotheses depends on the goals of the evaluation. Hypotheses can be evaluated using null hypothesis significance testing, where the null hypothesis assumes no effect or no difference (e.g. “CCAM has no impact on trip frequency”) and the alternative hypothesis posits an effect (“e.g. “CCAM increases trip frequency”). Researchers often use p-values to assess statistical significance. A p-value below a chosen threshold suggests that, assuming the null hypothesis is true, the observed effect (or one more extreme) would be unlikely to occur due to random variability in the data. This result typically leads to rejecting the null hypothesis. While very prevalent in scientific research, null hypothesis testing is not the only method for evaluating data in CCAM projects. Depending on the context, alternative approaches—such as constructing confidence intervals, focusing on effect sizes, or using Bayesian analyses—may be more appropriate. Example High-level research question: What is the impact of the CCAM service on people mobility? This can be refined into specific hypotheses: 1. CCAM service users’ monetary value to travel time is, on average, 20% lower than of the users of conventional human-driven vehicles. 2. The average number of trips per week is 10% higher for CCAM service users than for a comparable group of non-users. 3. The average trip distance for CCAM service users is 20% greater than that for a comparable group of non-users. 4. Within the operating area of the CCAM service, 50% of the trips that would otherwise have been undertaken using private cars are instead completed using the CCAM service. In practice, the drafters of the hypotheses should provide a justification for the specific percentages used based, for example, on theoretical reasoning or pilot data. Further reading FESTA. (2021). FESTA Handbook (Version 8). Chapter 6.1.1. https://www.connectedautomateddriving.eu/methodology/festa/ 52 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 3.3. Evaluation methods This chapter provides guidelines for selecting methods and tools to assess the impacts of CCAM. The primary goal of impact assessment is to estimate the magnitude of effects on specific indicators when a CCAM system is introduced or in use (treatment), compared to the situation before its introduction or when it is not in use (baseline). An effect refers to the relative difference in an indicator value that results from the use of CCAM. Effects can be expressed in quantitative terms (e.g. ±10%), qualitative terms (e.g. 'more', 'less', 'higher', 'lower'), or absolute terms (e.g. 100 injury accidents 13 , 1000 vehicle kilometres travelled, 20 tons of CO2). Careful planning is required to define what is being compared and under which conditions. Since the evaluation focuses in this handbook’s case on CCAM systems, their use and related impacts, it is important to isolate the changes caused by CCAM from other simultaneous changes and parallel trends, such as fleet electrification or transport service digitalisation. Therefore, the baseline and treatment conditions must be comparable and specified such that the assessment reflects only the effects of CCAM. It is also important to understand the differences between the CCAM system tested in field experiments and the CCAM system addressed in the impact assessment (see Chapter 3.1). Many CCAM field experiments involve systems with low technology readiness level (TRL), making it less meaningful to study the broader socioeconomic impacts of these early-stage implementations. Instead, impact assessments should focus on higherTRL versions of the systems, often assuming widespread adoption with a significant share of transport or mobility. These differences impose certain requirements on method selection, particularly in considering what can be directly measured from field experiments and how to bridge the gap between tested low-TRL systems and high-TRL systems targeted in the impact assessment. Once the research questions are established (see Chapter 3.2), indicators are defined to answer them. Indicators translate research questions into concrete, measurable metrics that can be evaluated using different methods. They bridge the gap between theoretical research questions and the practical, measurable dimensions of what those questions aim to capture. If hypotheses have been formulated, they may also influence the selection of indicators. The next step involves selecting suitable methods and tools for estimating the indicator values. The evaluation method specifies how each research question will be addressed after data collection from CCAM experiments. While some research questions might be answered using indicators that can be directly measured or calculated, others may require several stages with intermediary indicators before the primary indicator of interest can be assessed. Each step in this process may involve different methods, tools, and datasets to derive or assess interim and final outcomes. Scenarios are useful for structuring evaluations, as they enable detailed scoping and provide reference points for ensuring systematic coverage. In EU-CEM, a scenario is defined as a structured depiction, specified by a set of pre-determined conditions and variables, representing actual or theoretical states, situations, or interactions. All evaluation and impact areas require scenario setting, though the scope and detail of scenarios may differ across areas. 13 It is acknowledged that the terms ‘crash’ and ‘collision’ are often used interchangeably with ‘accident’. Projects may use any of these terms. 53 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility EU-CEM defines seven distinct scenario categories for evaluation (see Guideline 3.10 and Guideline 3.11 for more details): • Societal scenarios describe the broader societal context, including the envisioned CCAM system and its usage. • Service scenarios describe a single CCAM service or a chain of transport services that CCAM is part of, such as logistics services, public transport services, and shared mobility services. • Traffic scenarios represent traffic characteristics and road infrastructure on specific road segments or networks. Vehicles in a traffic scenario are involved in different, non-predefined driving scenarios. • Driving scenarios depict short, specific periods of driving defined by primary driving tasks (e.g. car following, lane change) or triggered by specific events (e.g. an obstacle in the lane) [13]. These describe smaller, more focused traffic contexts compared to traffic scenarios. They cover shorter periods of time and fewer road users. For example, interaction between two vehicles over a few seconds. In contrast, there may be hundreds or thousands of vehicles in an hour-long traffic scenario. • Test scenarios describe sequences of triggers, events, and actions among road users (e.g. ego vehicle, adjacent vehicles, pedestrians) to achieve a specific testing goal. This scenario type relates only to the evaluation of technical functioning (see Chapter 4.1). • User scenarios describe population segments and their state related to CCAM, such as mobility behaviour, attitudes, and socio-demographic characteristics. • User behaviour scenarios describe the direct or indirect interaction between different users and the CCAM system. While other scenario classifications exist (e.g. ISO 34501 [14] and ASAM OpenSCENARIO [15]), the one presented here is tailored to describe the diverse perspectives necessary for impact assessment of CCAM. In this handbook, traffic, driving, user, and user behaviour scenarios are considered as lower-level scenarios and societal scenarios as high-level scenarios. It is important to recognise that the scope of scenarios directly determines the scope of conclusions drawn from the results. Narrow scenarios, or a narrow set of scenarios, will only yield specific, context-bound conclusions. Broader societal conclusions require correspondingly broad societal scenarios. Field experiments typically address lower-level scenarios, such as driving or user scenarios, on a particular road section or within a limited area or based on a limited group of participants. However, broader societal impacts of CCAM require assessing them within societal scenarios that cover entire networks and all trips made in them, including non-automated vehicles and non-motorised travel. Depending on the impact areas and research questions, it may be necessary to scale up the impacts from lowerlevel scenarios to broader societal contexts reflected by the societal scenario. This requires a method for scaling up results to a societal level, reflecting a broader spatial and temporal frame than the lower-level scenarios. In addition to direct impacts, it is important to consider cascading indirect and possibly unintended impacts of CCAM. These include increased car usage due to improved comfort and behavioural adaptations among nonusers. These indirect impacts can be difficult to detect and quantify, as they may result from gradual processes. For example, behavioural adaptations among CCAM users and non-users can take months or years to fully manifest. Nevertheless, qualitative considerations of these impacts are recommended. In summary, this chapter guides the development of methods for answering defined research questions within the project’s timeline and available resources. It ensures that data collected by the project (see Chapter 3.4) is fully exploited during evaluation. Developing robust evaluation methods involves careful selection of suitable indicators, methods, and tools. This process is iterative, requiring coordination between the different elements of the evaluation plan described in Chapter 3 (see Figure 4). Once these iterations are complete, the final method is documented in the evaluation plan (see Guideline 3.28). 54 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Further guidance, specific to each evaluation and impact area, is provided in Chapter 4. This includes terminology definitions, background overviews, recommendations for result indicators, different methods and approaches, and best practices to avoid common pitfalls. These guidelines should be used alongside the general ones given here. The guidelines below cover the following topics: • How to describe the societal scenarios for impact assessment • How to define midand low-level scenarios: traffic and driving scenarios, user and user behaviour scenarios • How to define relevant indicators for impact assessment • How to set a method for answering a research question • How to consider the scaling up of results in the methodology • How to consider intended direct impacts, indirect and unintended impacts, and the impacts on nonusers These steps are interconnected and may require iterations; they are not strictly hierarchical or sequential. Note that the best chronological order of tasks may also differ from the sequence of guidelines below, depending on the project. Guideline 3.10. How to describe the societal scenarios for impact assessment The societal scenarios define the broader context and setting for impact assessment. They specify the region and timeframe of interest and describe how CCAM systems relate to existing transport infrastructure and supply. A societal scenario must be established for all baseline and treatment conditions planned for assessment (see Guideline 3.27). When defining societal scenarios, it is important to consider what requirements the research questions place on them. Evaluation can focus on one societal scenario or several alternative ones. Even if socio-economic or other broader societal impacts are not within the evaluation scope, defining a societal scenario is beneficial. Using shared societal scenarios across all evaluation and impact areas as a foundation for developing more relevant, targeted scenarios (see Guideline 3.11) ensures consistency. It ensures that results from different areas complement rather than contradict each by aligning assumptions across different evaluation steps or areas. If societal-level impact areas (see Chapter 4.4) are planned, the societal scenarios used in those assessments form the basis for the scenarios across all impact areas. The assessment of society-level impacts relies on results from other impact areas, which must align. To ensure feasibility across all impact areas, scenario development should be a joint activity. While maintaining consistency with the overarching societal scenario is essential, other impact areas include additional scenarios of specific interest. To define a societal scenario, set the characteristics of the high technology readiness level (TRL) CCAM system for which the impact assessment is made (see Chapter 3.1), the transport system it operates within, the CCAM users, and the targeted scope of evaluation in terms of time, place, and population. The process starts with the research questions, as their phrasing can specify the necessary scope of the societal scenario. If they do not specify all aspects, these must be agreed upon at this phase. The broadest defined scope sets the general context, such as “in the EU”, “on urban and rural roads”, “in city X”. The following table provides key elements to consider when defining societal scenarios. These elements provide a starting point, but are not exhaustive, and additional aspects—such as economic activity, political factors or socio-cultural trends—may be relevant. In the impact assessment, the impact of CCAM should be isolated from other changes in these elements. 55 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Element of societal scenario Examples of considerations Temporal scope • CCAM introduced into current transport system • Specific future year targeted (e.g. 10 years from now) • Specific period (e.g. from now until 2050) • Specific deployment milestone (e.g. 50% CCAM penetration rate in vehicle fleet) Spatial scope Specific city, region or country; all motorways in the EU Transport system and role of CCAM • Travel and transport service offerings and usage patterns • CCAM as a complementary service or disruptive replacement • Impacts on other transport modes • Fleet composition and vehicle characteristics • Systematic differences between CCAM and other vehicles, e.g. all CCAM vehicles being electric • Penetration rate of CCAM vehicles in traffic flow or vehicle stock CCAM target users • Car drivers, persons without a driving licence, certain demographic or social groups • Service and fleet operators, traffic management operators, professional drivers Other road users • Car drivers, public transport users, pedestrians, cyclists, micromobility users, certain demographic or social groups • Service and fleet operators, traffic management operators, professional drivers Dedicated infrastructure for CCAM • Dedicated lanes • Pick-up and drop-off zones • V2X and telecommunications equipment Regulations and policies • Need for a driving licence • Policies promoting shared mobility, electric vehicles or other • Taxation and insurance policies Example The project addresses one societal scenario with the following characteristics: Element Description CCAM system High technology readiness level as defined in Guideline 3.4 for the system addressed in impact assessment Temporal scope The current transport system (year 20xx with latest statistics available) 56 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Regulation and policies As in the temporal scope year Dedicated infrastructure to CCAM Infrastructure-to-vehicle connectivity available on motorways of the TransEuropean Transportation Network (TEN-T) to support temporary routing (e.g. roadworks) Transport supply Privately owned SAE Level 3 automated passenger cars replacing current privately owned conventional cars. Vehicles replacement occurs randomly, and the automated vehicles maintain the same size and powertrain as the replaced vehicles. No changes to public transport supply or freight transport compared to situation without automated vehicles. Targeted area All motorways in EU-27. Penetration rate of ADS among passenger cars in traffic 10% (short-term realistic) 50% (long-term realistic) 100% (theoretical maximum potential) Guideline 3.11. How to define midand low-level scenarios: traffic and driving scenarios, user and user behaviour scenarios The previously defined societal scenario serves as a basis for defining the mid-level service, traffic, and user scenarios, as well as low-level driving and user behaviour scenarios (see scenario definitions in the introduction to Chapter 3.3). The mid-level scenarios must align with the conditions set in the societal scenario, and the low-level scenarios with the mid-level scenarios. These scenarios add necessary detail for the impact areas that use them for evaluation, and several parallel scenarios can be used to study various conditions of interest. The selected midand low-level scenarios together should represent exposure (e.g. travel activity, vehicle kilometres travelled), road network, traffic patterns, fleet composition, as well as CCAM users, non-users, and their interaction with CCAM within the societal scenario. Although different evaluation areas focus on various impacts, their results should be complementary. Consequently, the chosen scenarios should not contradict one another across impact areas. The required number of scenarios depends on the evaluation scope, research questions, context addressed, and available resources. When balancing the depth of analyses with resources, the estimated effect size and exposure should guide scenario setting. It is advised to include scenarios where both positive and negative effects might occur. Relevant aspects to describe per scenario type include: • Traffic scenarios: • CCAM penetration rates, at least those required by the societal scenarios • Road environment and layout (intersection types, length of sections, number of lanes, speed limits) • Features of digital infrastructure (coverage and features of connectivity, etc.) • Traffic volumes 57 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility • Fleet composition in traffic (size, category, age of vehicles, engine and fuel types, pedestrians, cyclists) • Service scenarios: • Service concept • Business or funding concept • Vehicle category • Operating environment • Driving scenarios: • Vehicle whose behaviour is being addressed (ego vehicle) • Another vehicle or object to interact with • Road environment • User scenarios: • Mobility behaviour (motorised and non-motorised travel modes) • Attitudes • Socio-demographic characteristics • User behaviour scenarios: • User interacting with the system • Non-user affected by the system • Interaction with CCAM • Context for interaction (e.g. where it happens, in which conditions) Within one traffic scenario, several driving scenarios typically take place. Within one traffic scenario, several parallel service scenarios may also take place. One user behaviour scenario may apply to one or several user scenarios. If use of low-level scenario results is planned to address higher-level scenarios, it is advisable to plan it at this stage. This may involve scaling up to regional estimates or societal scenarios. Scaling up refers to estimating the impacts over a broader region and a longer time period compared to the lower-level scenarios. See the separate guideline on scaling up. If scaling up from low-level scenario results to higher-level scenarios is considered, this should be planned at this stage. Scaling up refers to estimating impacts over broader regions and longer periods compared to the lower-level scenarios (see Guideline 3.14). Example Example of the process for defining traffic scenarios: One impact assessment objective was to estimate the impacts of automated passenger cars on emissions. The societal scenario covered the European motorway network with 50% automated passenger cars in traffic, which included passenger cars and heavy goods vehicles. The baseline was the traffic and vehicle fleet of today. Traffic simulations and an emissions tool were chosen for estimating impacts. The societal scenario specified European motorways, passenger cars and heavy goods vehicles, and the penetration rate of automated vehicles among passenger cars. Representative traffic scenarios were derived to align with the societal scenario by: • Analysing the whole European motorway network to identify speed limit and number of lanes combinations covering at least 1% of the network length, resulting in 10 different combinations. • Determining traffic volumes by calculating motorway capacities (2250 vehicles/hour/lane) and dividing the volume range into four simulation input flows (500, 1000, 1500, and 2000 vehicles/hour/lane). 64 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility This ensures that the collected data 14 will support the evaluation. The data specification and data tools guidelines in this chapter are relevant if any of the cases below apply: 1. An evaluation method has specific requirements for its input data to be collected within the project, and the evaluation partner is not directly collecting that data. 2. It is planned that multiple partners will collect comparable data. 3. Data is shared between partners. The data specification and data tools step may not be as relevant when a dataset is collected by an evaluation partner to support their own work and is not shared among multiple partners, as the data harmonisation aspect is less critical in these cases. Data specification describes what kind of data the project collects, how it is formatted, and how it is stored. It ensures that data can be exchanged and used seamlessly among project partners. While data specification does not describe the experimental design of the data collection (see Chapter 3.5), it should describe contextual data that facilitate correct understanding of the design and usage and interpretation of the data. This includes whether the data was collected under baseline or treatment conditions, and situational variables relevant for the planned method (see Chapter 3.3). The data specification is the basis for a common data format, 15 which is used for storing and processing the data with common tools. Data collection tools refer to devices and programs used for data collection, such as data loggers installed in vehicles. In some cases, partners must agree on using common tools and how they are calibrated and configured to ensure comparable results. In other cases, simply harmonising the data format may be sufficient. Data processing tools refer to a chain of scripts or programs used to process the data. Processing includes converting logged data into the common data format and calculating derived measures. 16 A project may also need to agree on data storage and exchange tools. Examples of data storage tools are (federated) databases and data spaces which facilitate the exchange of data between stakeholders. Interfaces for uploading and downloading data may also be part of the tools. It is recommended that data specification and tools are clearly documented, either as part of the evaluation plan or in a dedicated data management document. Version control systems are highly recommended to ensure that every partner is using the same version of the specification and tools at all times. A documented data specification and a description of data tools is essentially an agreement between the data providers and the evaluation team on the technical characteristics of the data. The guidelines in this chapter do not mandate any specific data formats or data handling methods, leaving it for each project to decide. The CCAM Data Sharing Framework [18] gives detailed advice on data sharing, both from technical and contractual perspectives. 14 Collected data: Data collected with sensors, questionnaires or by other means. Also called acquired data. The collected data is also often referred to as 'raw data', but the term might be misleading, because some data processing might be performed by the sensors or other devices collecting the data. 15 Common data format: Provides the data specification in a manner that allows for computer-readable and writable implementation. It defines variables, their units, data types and other relevant specifications (see Guideline 3.17) 16 Derived measures: Measures not directly collected but calculated from the collected data. 65 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The guidelines below cover the following topics: • How to set up a process for defining data specification and data tools • How to define common data formats • How to define the roles and access rights related to data • How to define the data flow in the processing chain • How to overcome common pitfalls related to data These steps may be interconnected and require iterations; they are not necessarily hierarchical or sequential. Note that the best chronological order of tasks may differ from the sequence of guidelines below, depending on the project. Guideline 3.16. How to set up a process for defining data specification and data tools A documented data specification and a description of data tools is essentially an agreement between the data providers and data users on the technical characteristics of the data. To ensure that all perspectives are included, it is recommended that the evaluation team establishes a working group with representatives from partners responsible for providing, processing, and using the data. This group must also include partners working on the evaluation methods, not just data owners. Thus, the members of a data specification working group include: • Data user (evaluation team): Specifies the data needs of evaluation and understands dependencies between methods and data. • Data provider: Assesses the feasibility of providing requested data and under which conditions. • Data processor: Develops or uses scripts to process shared data into an agreed format. • Database developer: Implements and maintains a database for storing the agreed data. The process of defining data specification and tools is likely to be iterative. Use version control to track all changes, ensuring every partner is aware of what has been agreed upon, when, and by whom. Data specification and tools must be aligned with the evaluation plan. Therefore, the starting point for defining data specification is the data requirements identified for the planned evaluation methods (Chapter 3.3). It is important to: • Determine the data required (including combinations of data) and the contextual data (metadata) needed. • Engage in iterative discussions to balance what data providers can collect with what is required for calculating indicators and conducting evaluations. • Be flexible and open to alternatives if specific data is unavailable. On the other hand, sometimes available data collection or processing technologies may inspire new evaluation approaches. • Facilitate continuous dialogue between the technical team and evaluation team throughout the project to mitigate risks of failing to deliver suitable data for evaluation. Early dialogue helps align expectations and clarify data needs between providers, processors, and users. Request that data providers share an example dataset in the common data format to the evaluation team as early as possible. This ensures a shared understanding and verifies that the workflow functions correctly. Likewise, the evaluation team should provide an example of the preferred data format. Early conversion of collected data into the common data format can help identify inconsistencies between collected and required data. Harmonisation and agreement on the common data format and test data help data processing and evaluating partners to better anticipate what is achievable with the collected data. Therefore, test the data flow as soon as possible. 66 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility It is essential that the agreed data specification and tools are also communicated to all data providers, clearly outlining the requirements and providing the rationale behind specific data formats. For each data provider, identify the operational point of contact, ensuring someone familiar with the data collection process, specifications and tools is available to address issues. Once the agreements are established, any deviations from the agreed specifications should be documented and communicated. Example Data specification process: 1. Specification of data requirements: Identification of the necessary data for the planned evaluation methods, translation of the needed indicators and related contextual data into a list of signals to be collected, specification of the preferred data format. 2. Explanation of data requirements: Detailing what will be evaluated with the requested data and the consequences of missing data. 3. Discussion on feasibility: Assessment of whether the collected data and relevant contextual data can be shared in the specified format and under what conditions. 4. Iteration if needed: Return to step 1 if data cannot be provided (e.g. due to a lack of suitable sensors or critical data sensitivity). 5. Communication of agreements: Sharing of the final data specification and tools with all data owners, allowing for questions and feedback. 6. Refining if necessary: Revisit step 1 if critical issues are identified. 7. Provision of sample data: Sharing data samples in the agreed format to verify understanding and a functioning workflow, and to spot inconsistencies between collected and required data. 8. Maintaining dialogue: Ensuring continuous communication between data providers and users, documenting any deviations from the agreements until all data has been provided. Guideline 3.17. How to define common data formats A common data format provides the data specification in a manner that allows for computer-processable implementation. Various file formats can be used, including ASAM MDF, HDF5, XML or CSV, sometimes referred to as data formats. In EU-CEM, the file format is only one aspect of the common data format. The definition of the common data format includes at least the following: • File format • Name of variables • Description of variables • Data types of variables (e.g. integer, floating point number, text), size (if applicable) • Unit, frequency, and resolution of variables • Upper and lower limits of possible values of variables • Coordinate systems in which the variables are to be interpreted (e.g. the version of GPS coordinates, ego-centric or allocentric, direction of axes) • Timestamps used • Coding of missing values (e.g. NaN, -1) • Coding of categorical values (e.g. response options in a questionnaire, 1 = Yes, 0 = No) Establishing common file naming conventions is recommended to facilitate data usage. File names might include identifiers such as experiment identifications (IDs), driving scenarios, use cases or vehicle IDs. The common data format may also include details such as calibration values used in data collection or interpolation methods for handling missing values (e.g. zero-order hold, linear interpolation). For complex 67 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility data processing, it is advisable to develop and share common data processing tools (e.g. as scripts) to ensure consistency in data handling. Multiple common data formats may be necessary within a project. For example, one format may be used for collected data and another for the derived measures used in evaluation after the data processing chain has been applied. The CCAM Data Sharing Framework [18] gives recommendations on the file formats. Example An example is the L3Pilot Common Data Format [19], which includes the following elements: • Top Level (name) • Second Level (name) • Datatype • Size [byte] • Size on top level [byte] • Description (written description, explanation for the coding, e.g. 0 = baseline, 1 = treatment) • Unit • Frequency (Hz) • Resolution • Lower limit (minimum value) • Upper limit (maximum value) • n/a (coding of the missing values, e.g. NaN or -1) • Interpolation (the method which should be used to replace missing values, e.g. zero-order hold or linear) The L3Pilot Common Data Format can be implemented, for example, using the HDF5 file format. Guideline 3.18. How to define the roles and access rights related to data Defining the actors involved in the data flow17 clarifies responsibilities and ensures everyone knows who does what and who has access to the data. Typically, at least three roles are defined: • Data providers collect the data, for example through field experiments, participant interviews or simulations. • Data processors convert the collected data into usable formats for evaluation, such as by cleaning the data and calculating derived measures. • Data users utilise the data within the project; these may be members of the evaluation team. A single partner may take on multiple roles, but not all stakeholders need equal access rights, even when the data is stored in a common database. For example, data providers, and possibly a specified data-processing partner, may only have access to their own data. Some data may be too sensitive to share with the data-processing partners or data users. For example, data containing personal information may be subject to data protection regulations such as the General Data Protection Regulation (GDPR) [20]. Additionally, data may affect the business interests of partners if it is commercially sensitive and enables benchmarking the tested system against competitors. 17 Data flow: The process where collected data are transformed for and within the evaluation process through a chain of processing scripts utilising common data formats. 68 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The CCAM Data Sharing Framework [18] provides detailed guidance on establishing agreements between stakeholders in a project and on ensuring the privacy of the data. Example The roles assigned in a project with confidential data [20]: Each data-providing partner selected a dedicated data processor from the evaluation team. Each data processor had full access to the data from the respective data provider. This arrangement allowed the development and testing of common data processing scripts without requiring the data provider to repeatedly run the scripts on behalf of the evaluation team. The data-processing partners converted the collected data into the agreed common data format. Before sharing the data with the evaluation team, all processed data were pooled across data providers to prevent benchmarking between them. This role assignment ensured that the data were reviewed by someone other than the data provider without revealing potentially confidential data across all partners. In projects with non-confidential data, data providers can directly upload their data to a data sharing platform for processing and evaluation. Guideline 3.19. How to define the data flow in the processing chain The data flow process may include multiple processing and storage steps, and it should be thoroughly documented and reproducible. A key principle is ensuring that all analyses can be rerun easily if the original data are updated. In practice, this means all processing steps should be implemented as software scripts, avoiding manual modifications or interventions. Scripts should be maintained in a version control repository (e.g. GitLab) to ensure that all developers and users work with the same versions. Common databases or collaboration platforms can be used for data exchange among partners. It is recommended to split the processing into a series of modular scripts with clearly defined inputs and outputs, rather than using one large script. This simplifies maintenance, allows parallel development, and facilitates the creation of alternative scripts if necessary. If different partners are responsible for different parts of the data processing chain, intermediate outputs and inputs should be included in the data specification, with a common data format to ensure compatibility. Quality checks should be integrated into the data flow. These can include: • Data quality checks, such as visual inspection (i.e. “everything looks good”) or automatic checks to ensure that values fall within expected ranges and that the amount of missing data remains acceptable. • Script quality checks, such as unit tests for individual script functions and integration tests for combined modules, using both test data and real project data. These checks should flag any issues with the data or with the data processing before the data is used for evaluation. Filtering, such as signal smoothing, is often necessary during data processing and must be clearly defined within scripts, as it can significantly affect results. Consistent filtering criteria should be applied across all data sources, and data users must be informed of any filtering performed on the data to ensure correct interpretation. In some projects, additional processing steps before data sharing, such as anonymisation and pseudonymisation, may be required to avoid exposing the origin of the data or personal information of the participants. This step can be performed by data providers or by dedicated data-processing partners under 69 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility appropriate non-disclosure agreements if needed. Sometimes this is only necessary when data is shared externally. The Data Sharing Framework [18] gives detailed guidance on this. Example A dedicated working group maps the whole data flow of the project, detailing each step from data collection and conversion into a common data format to quality checks, filtering, annotation, and calculation of indicators. The indicator data are then stored in a cloud database accessible to all relevant partners with appropriate access rights. Next, each step of the mapped flowchart is established as a project in Gitlab, enabling the creation of small, manageable, and modular scripts to automate the data flow. Inputs and outputs of each script are explicitly defined, and version control ensures that only data processed with the latest script versions are accepted for subsequent steps. Quality checks cover both data and scripts, with unit and integration tests verifying functionality. Clear documentation for each script details its purpose, functionality, and usage instructions. The documentation is stored both as README files within the version control system and in a collaborative document management tool accessible to all project partners. This approach enables partners to apply the scripts regardless of their involvement in software development, ensuring data are shared in a harmonised format and processed in an automated, transparent, and reproducible way. The final processed data are stored in a cloud database, allowing the entire evaluation team to access them for evaluation purposes. Guideline 3.20. How to overcome common pitfalls related to data CCAM projects often encounter specific pitfalls related to data collection, processing and usage: 1) A derived measure required for evaluation cannot be calculated from the collected data. Involve the evaluation team early in the data specification process to allow them to specify what collected data or derived measures they need. The data providers should assess whether they can meet these requirements. Flexibility from both sides is required if adjustments are needed. See Guideline 3.16 for how to handle this situation. 2) Collected data lack sufficient contextual information on the data collection conditions for correct interpretation. Data specification must include contextual data such as weather conditions, road and infrastructure characteristics, traffic conditions, and user groups involved. This information is essential for selecting relevant data, understanding variability in the results, and determining the extent to which they can be generalised to other contexts. Document any deviations from the data specification and unexpected events during data collection, as they may influence data interpretation. For example, minor deviations from a designated route may not render the data unusable, but determination of this requires documenting the deviations. 3) Data collected or processed by different partners is not comparable. Ensure that all partners follow the agreed experimental design (see Chapter 3.5), use a common data format (see the Guideline 3.17), and apply shared, version-controlled processing scripts. 70 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Example Common pitfalls and solutions: Pitfall: The derived measure cannot be calculated from the collected data. • Example: The evaluation team requires a derived measure “distance to lane markings” to study the lane-keeping performance, but the field test vehicles do not detect lane markings and instead follow a pre-determined centre line from a high-definition map. • Solution: After discussions, the data providers agreed to share the vehicles’ deviations from the centre line. With an assumed lane width, the evaluation team estimated the distance to the lane markings from this information. Pitfall: The conditions under which the data were collected are not sufficiently known to ensure accurate interpretations when calculating a derived measure. • Example: Speed measurements show significant differences between days, but without contextual data (e.g. traffic conditions during the experiments) it is unclear whether the differences result from vehicle behaviour or external factors like congestion. Therefore, the measurements are less valuable for evaluation purposes. • Solution: Collect and document contextual data to ensure accurate interpretation of collected data. Pitfall: Data collected by different partners are not comparable. • Example: Data was collected on roads with different speed limits, or one experiment collected data only in free flow situations and another in congestion. • Solution: Harmonise experimental design and document all variations to ensure comparability. Pitfall: Data processed by different partners are not comparable. • Example: When filtering outliers, one partner applies strict conditions for a “valid” data point, while another filters only the clearly “wrong” measurements. This difference leads to different results. • Solution: Agree on standardised data processing protocols and filtering criteria across all partners. 3.5. Experimental design This chapter provides guidelines for designing appropriate experiments to collect the data needed to answer each research question. Selecting methods for data collection—whether through field experiments, simulations or other approaches—requires careful planning to ensure the validity and reliability of data. Key considerations include defining treatment and baseline conditions, determining sample sizes, selecting participants, and structuring data collection phases. Agreement on the experimental design is important for the feasibility of the evaluation plan execution in CCAM projects. Real-world testing opportunities are often limited to systems of low technology readiness level (TRL), making it impossible to accommodate every preference of the evaluation team. Some experimental design elements may have been established during the project preparation phase, particularly if they are also key project characteristics. This chapter focuses on selecting suitable design elements that can still be determined during the project. The experimental design must support the data collection required for each research question. If not, iterative adjustments to the evaluation plan are necessary (see Figure 4 and Guideline 3.8). For example, if a research question requires data from ordinary drivers but regulations prohibit such testing on public roads, alternative sources, such as test track experiments, should be considered. 71 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility A single CCAM project may involve multiple experiments, each potentially involving several teams, such as a technical team, a test site team, and an evaluation team. Early and consistent communication is key to ensure a feasible evaluation plan and efficient use of resources, and to avoid a mismatch between evaluation needs and the practicalities at test sites. Establishing standardised working procedures with clearly defined roles, responsibilities, goals and schedules early on is highly beneficial. Additionally, ensuring a common understanding through test site visits or direct discussions between evaluation and test site teams is recommended. The experimental design must specify the baseline and treatment conditions. In some cases, the baseline may reflect future rather than current conditions. Factors such as demographic shifts (e.g. aging population or more non-motorists), increased adoption of automation, and vehicle fleet electrification can significantly alter the results if unaccounted for. Predicting these shifts when defining a future baseline helps ensure that the experimental design captures the emerging impacts caused by CCAM only. The guidelines below cover the following topics: • How to choose the experimental approach for a research question • How to consider the elements of the experimental design in the evaluation plan • How to plan the experimental design of field experiments • How to plan the experimental design for virtual environments with participants • How to select participants for your experiment • How to provide participants with the most realistic user experience possible • How to define the treatment and baseline conditions These steps may be interconnected and require iterations; they are not necessarily hierarchical or sequential. Note that the best chronological order of tasks may also differ from the sequence of guidelines below, depending on the project. Guideline 3.21. How to choose the experimental approach for a research question Evaluating a single research question may utilise data from several approaches. CCAM projects typically use several approaches for data collection, including: • Field experiments (from limited to full CCAM system experience) o On test tracks o In confined zones o On public roads • Virtual environments, such as virtual reality or (mechanised) driving simulators. • Modelling and simulations, including microscopic or macroscopic traffic simulations, activity-based modelling, emissions models, and air quality and noise models. • Subjective approaches, such as written descriptions, user studies, expert consultations, and citizen engagement. Field experiments can be complemented with other approaches like virtual environments, modelling, and subjective studies to assess the potential impacts at higher CCAM penetration rates than currently possible in field experiments or to cover a wider range of conditions. Each method’s implications for the validity of the results must be understood. Field experiments can also complement other approaches, for example, by offering input for impact assessment simulations or other virtual environments to improve their realism. Ideally, field experiments would use CCAM systems of high technology readiness level in real-world conditions. This includes testing on public roads or in an environment where the future systems will be deployed, with potential future users of the system. However, since this is often not yet possible, alternatives like controlled environments, test tracks, simulated environments, or on-road tests with safety drivers or 72 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility operators are used. If local regulations permit public road testing, it is possible to collect data on driving behaviour and automated vehicle interaction with other road users in real traffic. Still, acknowledge that safety drivers are part of the safety protocol and have a different role and behaviour than future CCAM users. If real-world CCAM testing with real users is not possible, alternatives like test rides with vehicles driven by safety drivers, use of test-tracks, or in extreme cases use of storyboards, videos or written descriptions can be used to describe the system to participants, with opinions gathered through interviews, questionnaires or focus groups. The experimental design should ideally cover the CCAM system’s entire operational design domain (ODD) with sufficient variability in conditions. At minimum, it should cover key parts of the ODD relevant to the societal scenario addressed by the research questions. If comprehensive coverage is not possible, the evaluation plan must clarify which ODD conditions were included in the experiment and how the data collected reflect those conditions. See Chapter 4 for details on different approaches per impact area. Example Example research question: What is the impact of CCAM on the accident risk? The consortium initially considered collecting real-world data from automated passenger cars on public roads. However, due to the safety protocols of the data providers, only safety drivers were permitted, and safetycritical situations had to be avoided. Thus, the field experiments alone would not provide sufficient data for a thorough assessment of the impacts on accident risk. To overcome this, simulation of the safety-critical events was identified as the main method to complement the field experiments. Field data would provide information on incident frequency in regular driving conditions and support the parameterisation and validation of the simulation model, which would then assess accident risk across various driving scenarios. Guideline 3.22. How to consider the elements of the experimental design in the evaluation plan The experimental design consists of several elements, each of which must be considered during evaluation planning. Research questions and data needs determine the essential (must-have) and desirable (nice-tohave) requirements for each element, alongside external constraints such as: • Local and national regulation related to location, driver type, timing, border-crossing, permissions, insurance constraints or data protection • Ethics and safety procedures • Available resources, including equipment, budget, time, people, and skills • Technology readiness of the CCAM system Each element should be optimised to fulfil the research question requirements within these constraints. Backup plans should be developed, and the impact of alternative choices on results validity and applicability must be assessed and documented. Describe the non-ideal factors and how they affect the results in the report that describes the evaluation method. Iteration between experimental design, evaluation methods, and data specification is necessary until a suitable evaluation plan is achieved. The table below lists the elements of the experimental design and their consequences on the evaluation. The experiment is a Field Operational Test (FOT) when all its elements are on the more complex or larger scale of the range. 73 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Element Range Consequences on evaluation Sample size Small to large Smaller sample sizes in terms of e.g. number of participants, observations, vehicles, test environments, or scenarios, allow in-depth analysis of complex interactions. Larger samples provide more generalisable results and statistical power. Power analysis can determine appropriate sample size. Sample demographics Limited to diverse Limited demographics allow focusing on specific user segments. Diverse demographics enable population-level estimates. Nature of data Subjective to objective Subjective data are often qualitative, more open to interpretation, and time-sensitive, but can provide rich insights. Objective data are often quantitative, consistent, and repeatable. They can be more constrained by CCAM TRL than subjective data. Data measurement Proxy to direct Direct measurements are preferable, but proxy measurements can be used when direct data are unavailable or unreliable. Proxy methods must be well-documented (see Chapter 3.4). Degree of realism of the experience Written description to real-life experience of CCAM use Higher realism (e.g. real-world tests) improves reliability but depends on technology readiness. Virtual environments (simulators, virtual reality) can mimic reality but have limitations in participant responses. Written descriptions offer lowest realism but can be easily provided to large populations. User experience in field experiments Single demonstration ride to non-supervised automated vehicle use Safety protocols limit testing possibilities. Ideally, participants would use CCAM unsupervised over longer periods. However, sometimes it is only possible to provide single demonstration rides. These offer limited but valuable insights. Experimental environment Short fixed test route to extensive road network Short, fixed test route limit generalisability. Extensive road networks provide broader insights under varied conditions. Technology readiness level (TRL) Low to high TRL Low TRL may not fully reflect the future system but can offer preliminary insights. Gradual testing by incremental pilot phases can mitigate the gap between low-TRL technology tested in the real world and a high TRL assumed for impact assessment. Likely requires a sequence of projects. 80 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Guideline 3.27. How to define the treatment and baseline conditions The baseline serves as the reference point for comparison with the treatment condition during impact assessment. Defining both conditions and determining indicator values for them is essential. The treatment condition refers to the CCAM system operating in a defined environment or context, while the baseline condition refers to the situation against which the impacts of CCAM are assessed, such as a situation without the CCAM system. Baselines can be established in several ways: • A baseline without CCAM compared to a treatment with CCAM. • A current operational design domain (ODD) baseline compared to an extended ODD treatment. • Multiple baselines for comparing different levels of automation. Treatment conditions can be set in several ways, such as varying CCAM penetration rates or with multiple CCAM service variants. The experimental design should ensure that the planned differences between treatment and baseline conditions are the only differences present, avoiding other systematic differences which might bias the results. In field experiments, the baseline and treatment periods should ideally be of equal length and in similar conditions. This allows the baseline period to capture the same variations that occur during the treatment period. These variations could concern seasonal effects, time of day or inexperience of the test subjects. Collecting more data for both baseline and treatment conditions strengthens result robustness. Conducting a statistical power analysis prior to data collection helps determine the required volume of data for reliably detecting a specified minimum detectable effect size. Rough estimation of the effect size can be based on prior studies, preliminary data or domain expertise. Pre-existing data, like naturalistic driving studies from prior projects, might serve as viable baselines in some specific instances. However, ensuring that the existing data is comparable to the treatment data collected in field experiments requires careful consideration of the vehicle, traffic environment, environmental conditions, driver population, data logging details, and so on. There are several ways to set the baseline and treatment conditions of societal scenarios. Multiple alternative baselines may be suitable. Choosing an appropriate baseline and treatment conditions for societal scenarios requires balancing between realism, workload, available resources, and the uncertainties involved. A present-day societal scenario requires that both baseline and treatment conditions reflect the current situation in all respects except for CCAM presence. This involves considering the region’s travel patterns and vehicle fleet using statistics, travel surveys, and other similar sources. Although assuming widespread CCAM adoption today may be unrealistic, it eliminates the need for uncertain predictions. Societal scenarios employing a future baseline can be more complicated, as they require forecasting the development of traffic systems and travel patterns for the specified future period. Be aware of the consequences of the baseline choice. For example, predicting future baseline conditions adds workload if suitable predictions are not available from existing sources. Each impact area and data type may require specific baseline definitions, such as: • Accident data: Number of accidents in traffic without automated vehicles. • Traffic data: Total vehicle kilometres travelled in the region by the current vehicle fleet • Energy and environment impact area: Engine types for emission calculations as predicted for year 20xx. • Mobility impact assessment: Public transport service offering in the area before the introduction of CCAM. 81 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility A baseline may not be required when only observing behaviour or technical performance, like in user evaluation or technical functioning. Expertise is required to select the relevant elements of baseline and treatment conditions for each research question and impact area. Example Case #1: Automated shuttle bus service in city centre • Treatment: Automated shuttle bus service transporting people between public transport hubs and points of interest in the city centre. • Baseline: Current mobility services in the area. Case #2: Automated shuttle bus service in a newly built business park • Treatment: Automated shuttle bus service in a newly built business park. • Baseline: No direct baseline can be collected from the business park, as it is new and has had CCAM since its completion. Alternatives include performing observational studies or finding comparable data from similar business parks. Case #3: SAE L4 motorway automated driving for passenger cars • Treatment: Passenger cars with motorway ADS of SAE L4 at 10% penetration of traffic flow. • Baseline A: Traffic flow of human-driven cars, no automation. • Baseline B: Traffic flow of human-driven cars with adaptive cruise control (ACC). • Baseline C: Traffic flow of 10% ACC-equipped and 90% non-automated cars. Case #4: Societal scenario: Automated passenger cars • Treatment: 20% automated cars in traffic, projected 10 years into the future. • Baseline: Traffic prediction for same period without CCAM. 3.6. Evaluation plan compilation This chapter provides guidelines for compiling an evaluation plan, which integrates all the steps of the planning phase: CCAM system description (Chapter 3.1), research questions (Chapter 3.2), evaluation methods (Chapter 3.3), data specification and tools (Chapter 3.4), and experimental design (Chapter 3.5). Given the multiple iterations involved in the planning process, it is recommended to compile the formal report on the final evaluation plan only after completing the iterative process and establishing a feasible plan. This prevents unnecessary revisions when making adjustments during the planning phase. The evaluation plan serves as a guide for the evaluation work throughout the project. It is recommended that the evaluation plan be published as a public report of the project, specifying the final list of research questions and the methods for answering them. This facilitates dissemination of the planned evaluations and allows for concise method descriptions in the evaluation results report. Despite thorough planning, adjustments to the methodology may be necessary during its execution. It is important to document any changes and the rationale behind them in the evaluation results report, particularly if the evaluation plan has already been published. This ensures that the final methodology is accurately documented for future use and correct interpretation of the results. If the evaluation plan is included in the final results report, only the executed methodology is published. Any deviations from the EU-CEM guidelines, along with their justifications, should be specified in the reports. Additionally, explain the reasoning behind the selection of evaluation and impact areas. 82 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The guidelines below cover the following topics: • How to compile the evaluation plan • How to describe the linkages between different evaluation and impact areas • How to ensure resilience of the evaluation plan Note that the best chronological order of tasks may also be different from the sequence of guidelines below, depending on the project. Guideline 3.28. How to compile the evaluation plan Describe the decisions and outcomes of all elements of the evaluation plan (Chapters 3.1 to 3.5) in a concise and coherent report. Briefly explain the rationale behind decisions and assumptions, including references used or expert assessment made within the project. Below is a suggested outline for the evaluation plan report: • Overall aim and scope of evaluation, including impact areas covered and a description of the CCAM system under evaluation (see Guidelines 3.1, 3.2, 3.3, 3.4). • Experimental design(s) applied. • Societal scenarios in both baseline and treatment conditions (see Guidelines 3.10, 3.27). • For each evaluation and impact area: • Overall approach and how the common scope is applied in this context (see Guidelines 3.5, 3.13, 3.14, 3.15). • Research questions and indicators used to answer them (see Guidelines 3.6, 3.7, 3.8, 3.12). • Evaluation methods for each research question, including tools, experimental design, input data, and the scenarios addressed (see Guidelines 3.11, 3.13, 3.14, 3.15, 3.17, 3.23, 3.24, 3.25, 3.27). It is advisable to draw a timeline of the different evaluation phases, such as a Gantt chart for internal project use, though it does not need to be published. Since developing the evaluation plan involves iterative refinement, the report should include only the final specifications and their justifications. When reporting the results of the evaluation after its implementation, document any changes made to the original plan and the reasons behind them (see Chapter 5.1), especially if a separate evaluation methodology report was published earlier. Final methodological details are often refined during implementation and must be included in the evaluation results report. Example Example outline of an evaluation plan report: • Overall scope of evaluation and impact areas covered • Description of the CCAM system(s), ODD, and service concept • Common societal scenarios in baseline and treatment conditions • For each evaluation and impact area (where relevant): o Overall approach and scope, including what impacts are in and out of scope. o Research questions and indicators used to answer them o Experimental design per research question o Evaluation method per research question, including tools, input data, and relevant traffic, driving, user, and user behaviour scenarios o Data formats used • Comply or explain—describe deviations from the EU-CEM guidelines and their justifications. 83 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Further reading Innamaa et al. (2020). Evaluation plan. L3Pilot Deliverable D3.4. https://l3pilot.eu/fileadmin/user_upload/Downloads/Deliverables/Update_07102021/L3Pilot-SP3-D3.4Evaluation_plan-v1.0_for_website.pdf Vater et al. (2023). Effects evaluation methods. Hi-Drive Deliverable D4.5. https://www.hidrive.eu/app/uploads/2023/09/Hi-Drive-SP4-D4.5-Effects-evaluation-methods-v1.0-.pdf Guideline 3.29. How to describe the linkages between different evaluation and impact areas Each research question requires its own evaluation, but synergies often exist within or across impact areas. Multiple research questions may rely on the same input data, while some require outputs from other impact areas as input. Identifying and accounting for these interdependencies early ensures consistency of evaluation results and a coherent, timely evaluation process for all impact areas and research questions. Begin by listing the necessary input data and expected output indicators for each evaluation and impact area, considering all relevant methods and tools. A visual sketch of these linkages is recommended to help illustrate the processes and dependencies. For each impact area, specify whether the required inputs come from within the project—either directly produced or collected—or from outputs or interim results of other evaluation and impact areas. Identify any external sources if applicable. Clearly define the expected contributions from each partner and the tools involved. When compiling the evaluation plan, consider the time required for each step in the methods used to produce results for each evaluation and impact area. Allocate sufficient time for each step, especially when an evaluation area depends on outputs from another area. Both areas need adequate time for tool setup, data collection and processing, analysis, validation, and feedback loops. Avoid overly tight scheduling to accommodate potential delays or adaptations in methods or tools. Identify critical steps early and set additional internal milestones for their completion. These internal milestones are complementary to project-level milestones and serve to monitor progress. Example Linkages between impact areas: • Driving behaviour impact assessment: Evaluates the impact of driving automation on the average headway during car following. o Analysis of headway differences between manual and automated driving using field experiment data. • Traffic flow efficiency impact assessment: Evaluates the impact of different headways on total delays in a region. o Simulations estimate the impact of headway differences on traffic flow at high penetration rates, using headway results from driving behaviour assessment when setting up the simulations. • Socio-economic impact assessment: Monetises total delay impacts for use in cost-benefit analysis. o Traffic-flow impact results are monetised and used as input in cost-benefit analysis. 84 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Guideline 3.30. How to ensure resilience of the evaluation plan Resilience of the evaluation plan refers to its ability to withstand minor changes during the project without requiring significant adaptations. A resilient plan minimises reliance on single experiments, development of new tools, individual team members, or tools available to only one partner. Feasibility and resilience are distinct concepts: a plan can be feasible without being resilient, or resilient without being feasible. Both should be assessed separately. The evaluation plan should be designed to accommodate unforeseen challenges, including issues with data collection, changes in project personnel, and timeline adjustments. To strengthen resilience: • Involve multiple team members in each topic area, even if one plays a smaller role, to ensure continuity in case the primary responsible person leaves the project. • Establish contingency plans for all new developments, using existing tools and methods as backup, even if they yield less reliable results or require adaptations, such as reducing the number of scenarios or altering result indicators. Additional recommended measures include: • Assigning clear responsibilities early on, ensuring that all partners can plan necessary resources for method and tool development. • Appointing responsible persons for each evaluation and impact area, research questions, and phase of the evaluation. Assign them the task of timely plan development and execution. • Verifying that all partners have sufficient resources (time, budget, tools) to meet their commitments, and conducting risk analysis of method and tool development to assess different levels of success and consequences of failure. • Piloting the methodology in advance to confirm that each step works as intended, that outputs align with subsequent needs, and that all required data sources are verified. • Agreeing on the priority of research questions and scenarios, while ensuring that high-priority items are addressed first in case the project runs out of time and evaluation cannot be completed. • Presenting the methodology to the consortium, especially the research questions and data needs, and finding solutions to any potential disagreements. Maintain dialogue throughout the process, but ensure that discussions take place before finalising the evaluation plan. 85 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility • Ensure that necessary ethical approvals, legal requirements, and risk assessments are in place. Although this may be the responsibility of the test site team, the evaluation team should verify this. Example Context: The project aims to study the impact of automated driving on car sickness using driving simulators. Potential risks identified: Reliance on a single partner for simulator adaption; the motion sickness measurement tool might break or provide noisy data; a single data source for physiological measurements; limited time for data collection and analysis; high variability in participant susceptibility to car sickness. Measures implemented: • The consortium selected two partners with access to a driving simulator, with three additional partners collaborating in adapting the simulator software to the project’s approach, allowing one to take over if another faced delays. • Critical tasks were identified and backup partners assigned, and deadlines for the main tasks were set. Linkages within the project were explicitly established. • A questionnaire was developed as backup for measuring car sickness if the primary tool failed. • Additional physiological measurement devices were installed as proxies for car sickness. • An early pilot test identified and addressed issues before the main data collection, ensuring adherence to the tight schedule and allowing for better control of simulator settings and participant variability. 86 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4. Evaluation-area-specific guidelines This chapter expands the generic guidelines for the development of the evaluation plan (see Chapter 3) with evaluation-area-specific guidelines. These 18 areas cover evaluation at the level of a vehicle, a human, a transport system, and the entire society (Figure 5). For each evaluation area, the following is provided: • Introduction o Definition for the evaluation area and scoping of the chapter. o Background with explanation of the key concepts, how CCAM relates to these impacts and a short overview of the theory where applicable. • Guidelines o Recommendations for common result indicators to be calculated by all CCAM projects. (A full list of recommended indicators is provided in Appendix with links to the corresponding chapters.) o Impact pathways that show the linkages with other evaluation areas. o Overview of different approaches and methods with their pros, cons, and requirements. o Pitfalls related to the methods and approaches and actionable best practices to address them. The linkages between different evaluation areas are visualised in the impact pathways, which show aspects leading to the impacts of CCAM in each specific evaluation area. Figure 5. Evaluation areas addressed in Chapter 4. 87 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Sustainability is placed under other impact areas as a cross-domain (Figure 5) which takes a holistic look at the impacts found for the environment, society and the economy, as most of the impact areas described in this handbook relate to it. EU-CEM makes a strong recommendation that all the projects reflect on their results from the viewpoint of sustainability, regardless of the impact areas chosen for evaluation. The guidelines for sustainability impact assessment can be found in Chapter 4.4.6. 4.1. Vehicle 4.1.1. Technical functioning 4.1.1.1. Introduction Definition and scope EU-CEM defines technical evaluation as follows: Technical evaluation addresses the functioning of the software and hardware of a CCAM system. In EU-CEM, technical evaluation is distinct from the technical validation and verification of CCAM systems and testing processes, such as data logging. Validation and verification are assumed to have been completed by the system developer or a third party prior to technical evaluation in EU-CEM. It is also assumed that the CCAM system has been built, tested, and is either deployed or deployment-ready before its use in the project. In this chapter, CCAM system refers to either a vehicle or a system installed in or used by a vehicle. Consequently, the primary focus of the technical evaluation is the vehicle itself. However, technical evaluation may be conducted in stages—initially evaluating individual components and later expanding to evaluate the entire system as an integrated whole. Whenever this chapter references the evaluation of a CCAM system, it simultaneously pertains to the evaluation of its individual components. Thus, the technical evaluation focuses on: • CCAM systems installed in vehicles: For example, a newly developed and tested system integrated into a commercial or pre-existing vehicle. • CCAM vehicles themselves: For instance, a new entire vehicle prototype. Where relevant, the evaluation also includes the functioning of vehicle-to-infrastructure connectivity and vehicle-to-vehicle cooperation. User-related considerations are excluded from technical evaluation and are addressed in Chapter 4.2.1. Although technical functioning may have implications on driving behaviour, such aspects are addressed in Chapter 4.1.2. Therefore, the scope of this chapter includes: • Evaluation of a built, tested, and deployed or deployment-ready CCAM system. • Objective and quantitative measures of technical parameters that address specific research questions. • Guidance for evaluating technology-related aspects. The scope of this chapter does not include: • Pre-deployment verification and validation processes. • Subjective or qualitative measures on perceived technical functioning. • Driving-behaviour-related measures (see Chapter 4.1.2) • User-related evaluation, such as user interaction with the technology or service (see Chapter 4.2.1). 88 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Background Technical evaluation focuses on the technology that makes the CCAM system functional, while functional evaluation examines the system’s ability to produce the expected outcome under defined conditions and specified inputs. Additional perspectives may include performance-related evaluation, which investigates: • How effectively the system produces the expected result. • The system’s tolerance—its ability to generate consistent results under varied conditions. • Its sensitivity, that is, the degree to which changes in input affect the output. Given the variability in complexity and technological readiness among CCAM systems, technical evaluation employs diverse approaches and methodologies. Ideally, it assesses objective indicators—whether numerical, categorical, or otherwise—that quantify a certain amount, value, quality or consequences of the system relative to the research question. Overall, technical evaluation is understood as a series of structured and systematic tests. Technical evaluation shares conceptual similarities with the verification and validation [23] processes in systems development, yet there are subtle but notable differences: • Verification is an internal process conducted by the system developer during the development phase, with the goal to evaluate the accuracy, robustness, and other technical aspects of the system. Verification often responds to the developer’s question, “Are we building the system right?” • Validation is an external process carried out during the testing phase by the system consumer, with the goal of evaluating the validity of the system against defined user needs. It responds to the user’s question, “Are you building the right system?” • Technical evaluation in EU-CEM is an external process conducted during the operational use of a built, tested, and deployed or deployment-ready CCAM system to evaluate the value, performance or other objective measurements of the system. It responds to the research question: “How does this CCAM system perform?” Although the purpose of technical evaluation of EU-CEM differs from that of verification and validation, similar approaches and concepts can be applied. Without going into the specifics of software, hardware or other service-level processes, common evaluation approaches include: • Scenario-based evaluation: The preparation of specific, manually created, purpose-built or preproduced definitions of the environment, actors, expected sequence of actions, triggers, conditions, and other contextual information that constrains the evaluation to a controlled domain. These test scenarios may be derived from expert knowledge, accident databases (e.g. accidentology or dedicated scenario databases) or other documental sources. • Data-driven evaluation: The utilisation of data generated during the operational execution of the system, or from an equivalent reference system, to evaluate its technical functioning. Each approach has advantages and limitations. Data-driven evaluation tends to be more reflective of real-world conditions, since it captures live operational data; however, it may fail to capture infrequent but critical edge cases. Such edge cases can be challenging to encounter in real-world conditions due to their rarity, the expense or danger associated with replicating them, or ethical and legal constraints, implying unreasonable risks for the test site team or participants, especially pedestrians or other vulnerable road users. In these cases, scenariobased tests can efficiently target these edge cases, albeit with simplified representations of reality. A combined approach—integrating both scenario-based and data-driven evaluations—may balance these benefits and drawbacks. Nevertheless, no single approach can entirely eliminate or fully quantify the gap between the evaluation stage and real-world functioning. Edge cases, by their very nature, are often unknown 89 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility because they have not occurred before, are difficult even for experts to imagine, or are too complex or rare to be thoroughly represented in current models. The testing method used in the technical evaluation is an important consideration. In both scenario-based and data-driven approaches, various testing methods can be applied to achieve the same goal. Current verification and validation approaches often combine the following types of test methods either sequentially or in a customised manner: • Virtual testing: Tests are conducted on a simulated system that employs virtual environments, virtual models of vehicles, and simulated actors. Virtual components may run independently or be integrated into a single simulation framework. In the latter, the tests follow a method or test definition procedure that establishes all the contextual information and captures the expected output variables to produce test results. • X-in-the-Loop: The generalisation of specialised test methods such as Model-in-the-Loop, Software-inthe-Loop, Hardware-in-the-Loop, Driver-in-the-Loop, and Vehicle-in-the-Loop [24]. X-in-the-Loop implies that part of the evaluation is virtual while other parts are real (e.g. physical or in their final form). This requires specialised equipment or facilities that connect the virtual and non-virtual components. Representative tests for validating technical functioning require their natural coexistence. For instance, Vehicle-in-the-Loop systems connect real physical vehicles to a mechanism that mimics the road, environmental conditions or other aspects the vehicles may encounter in reality. Similarly, Hardware-in-the-Loop connects a physical device to interfaces for receiving simulated data, mimicking its actual deployment environment (e.g. an electronic control unit in a vehicle) [25]. • Proving ground: Controlled real-world settings where CCAM systems can be technically evaluated without facing live traffic situations. Controlled test tracks are closer to reality than virtual testing or Xin-the-Loop, but the tests are limited by the test track capabilities. Proving grounds can be prepared to emulate varied environmental and contextual conditions. These include different types of roads, dummies simulating real pedestrians, effects of water, rain and dedicated wireless communication networks. • Real-world testing: The ultimate testing method for CCAM systems prior to their large-scale deployment in the real world. Real-world testing implies dedicated pilots or field operational tests (FOTs) that require careful preparation of permissions, analysis of routes, involvement of other actors, traffic conditions, and so on. One or more of these testing methods may be necessary depending on the research question. Harmonisation of results across different methods is a challenge, as the results may vary per method, reflecting the gap between the testing method and the expected technical functioning of the system in reality. The harmonisation should address these gaps, either by trying to decrease them, trying to understand and interpret them, or at a minimum being aware of their magnitude. Finally, technical evaluation can also be framed within the context of different technology development models: • V-Model: Often associated with waterfall approaches, where user needs guide the requirements and specifications for development. This is followed by testing, including unit and system tests, and validation of user needs. In EU-CEM, the system is assumed to be already built and tested, so the VModel may be applied to produce technical evaluations by developing and verifying the required components. • Agile (iterative): A cyclical or iterative approach without beginning or end, with cycles of continuous development, testing, deployment, and monitoring phases, which can accommodate rapid adjustments and iterative improvements. 96 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility • Frequency of and duration spent in specific driving scenarios: how often and for how long the automated vehicle engages in certain driving scenarios. For example, whether the vehicle makes fewer lane changes, more cut-ins occur, or the vehicle spends more time following a lead vehicle. Not within the scope of this chapter are: • Evaluation of the system’s technical functioning (see Chapter 4.1.1), as the vehicle is expected to be functionally working as intended. • Evaluation of human driving behaviour except as a baseline, or human driver states and traits (see Chapter 4.2.1). • Interaction between the automated vehicle and entities beyond its immediate vicinity, including vehicles and other road users. • Traffic flow efficiency (see Chapter 4.3.5). • Accident 18 investigation following an incident during real-world testing. Background Driving behaviour is often described as the result of interactions between the driver, the vehicle, and the environment (see Figure 8). In automated vehicles operating in automated mode, the driver is the automated driving system, and in manual vehicles, the driver is a human. Automated driving systems are anticipated to exhibit different driving behaviour compared to human drivers. Furthermore, different systems may vary in their characteristics or requirements, such as the operational design domains (ODDs). Automated vehicles may also differ from manually driven ones in terms of dimensions, mass, and equipment such as sensors and actuators. Relevant environmental aspects include infrastructure requirements and driving conditions, such as weather and traffic flow status. 18 During the writing of this handbook, it was noted that the terms ‘a crash’ or ‘a collision’ were often used interchangeably with ‘an accident’. It is perfectly acceptable to use any of these terms. 97 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 8. Main components of the driving behaviour impact area. Longitudinal and lateral movements of vehicles can be observed in the real world, or in virtual environments such as driving simulators. Determining the context around the movements is necessary for correct interpretation of the indicators and for justified comparisons. For example, comparing merging manoeuvres from a ramp onto the motorway with other mandatory lane changes on the motorway might be misguided due to different contexts in terms of urgency of the manoeuvre, even though both feature a mandatory lane change to the left. The contextualisation requires examining trajectories, calculating indicators, and collecting data on vehicle properties, surrounding objects, and the environment (e.g. road infrastructure, weather). The data should allow for segmentation of driving scenarios. This can require knowing, for example, the road type, prevailing speed limit, traffic state (free flow vs. congestion), and weather conditions. In addition, the data can include human driver traits and states, but this is not covered by this chapter. It is necessary to log the mode of the vehicle at every moment: manual driving, manual driving supported by ADAS, automated, or connected and automated (e.g. using vehicle-to-everything) or transitioning between these modes. These variables should also be logged in simulations, depending on the simulation complexity and availability of data and models. Automated vehicles can influence the behaviour of other road users, for example by leaving larger gaps for other vehicles to cut in to due to minimum headway settings. The implications of these interactions for traffic flow efficiency are considered in Chapter 4.3.5, but they are not limited to changes in efficiency. For example, changes in the frequency of very short gaps can affect the number of critical situations. Note that for setting the driver models for the simulations in other impact areas, input is likely required from driving behaviour analyses. 98 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4.1.2.2. Guidelines Analysing and explaining field observations of driving behaviour requires understanding of the context around the observations. This includes the technological readiness level of all systems, the technical functioning of the CCAM system (see Chapter 4.1.1), considering the type of participants involved (e.g. are participants in the vehicles professional safety drivers or members of the general public?) and the openness or restrictions of the environment. Determine who is in control at each moment during a trip—the human or the ADS. That is, log whether the vehicle is in automated or manual mode at each point in time. In addition, identify the moments when control shifts between the modes. If remote operation of the ADS is involved, its status should be logged as well. In virtual environments, it is important to know the limitations of all driver models. Determine which parameters are defined already in the automated vehicle model, and which parameters can be assessed in the virtual environment. For example, identifying the target headway of an automated vehicle during car following by simulation is redundant when the headway is a parameter in the simulation configuration. Rather, evaluate aspects arising from the changed parameter settings of the simulated vehicles, such as the frequency of cut-ins and lane changes in different traffic compositions. Driving scenarios should be detected and segmented from the data on the ego vehicle’s longitudinal and lateral movements and its surrounding objects. Driving scenarios can be predefined or staged in controlled and virtual environments, or they can result from triggering events or interactions. Examples of events are harsh braking of a preceding vehicle, a request for take-over of control, an obstacle ahead, and a vehicle cutting in. An event can also be an interaction between the automated vehicle and an emergency vehicle or a human traffic controller overriding traffic signals or signs. These events can be predefined, generated or occur naturally during an experiment. When using real traffic data, the events and, thus, the driving scenarios need to be detected during post-processing. Indicator recommendations Table 5 lists the indicators recommended to be evaluated by every project addressing driving behaviour. They cover the main features of driving behaviour needed for simulations (speed and headway) and proxies for energy and environmental (acceleration sum) and traffic safety (harsh breaking) impacts, as well as other changes in traffic flow dynamics, such as frequencies of different driving scenarios. In addition, each project should define indicators of their own in accordance with Guideline 3.12. Indicators should be calculated, when relevant, as both absolute and relative values (e.g. per vehicle kilometre). Impacts are differences or changes in these indicators between the baseline and treatment. Distinguish each indicator per relevant categories of situational variables, such as road type, traffic state, and weather conditions. For example, studying the average speed per speed limit can additionally focus on calculating it under different weather conditions. The ´comply or explain´ principle of EU-CEM necessitates reporting reasons when some indicators are not evaluated. Table 5. Indicators recommended to be evaluated by every project addressing driving behaviour. Indicator short name Definition Unit Average speed per speed limit Average speed of the ego vehicle during free driving, categorised by speed limit. If privacy is a concern (e.g. the speed limit may reveal the vehicle’s location), the percentage difference between the vehicle’s speed and the speed limit can be calculated instead. [21] m/s or km/h 99 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Median headway during car following The median value of the ego vehicle’s headway (in seconds) to the preceding vehicle in car-following situations. Seconds and % Acceleration sum Sum of positive longitudinal and lateral accelerations per vehicle-km, measured separately per driving scenario. Longitudinal accelerations allow studying forward motion behaviour, such as acceleration or deceleration patterns. Lateral accelerations help evaluate, for example, lane keeping during free driving and car following and handling and stability during side-to-side manoeuvres like lane changes or intersection driving. m/s2/vehicle-km Frequency of harsh braking events Number of ego vehicle decelerations exceeding a threshold of 5 m/s², measured per distance driven. Number/vehicle-km or number/h Frequency of and/or % of distance driven in driving scenarios Both frequency and distance driven are relevant for several driving scenarios. For example, for lane change scenarios the frequency (number of occurrences per distance driven) is applicable. For prolonged driving scenarios (e.g. car following), the proportion of distance driven in that driving scenario provides more value. Number/vehicle-km or % (of distance) per driving scenario Data is assumed to be available on the automated driving mode status and whether tele-operation occurred. Additional indicators relevant for specific driving scenarios may include the number of take-over requests, position in lane, number of unscheduled or unnecessary stops, speed variation, and other indicators of longitudinal and lateral driving behaviour. For more dynamic driving scenarios (e.g. a braking lead vehicle or when approaching slower traffic), recording conflict measures such as minimum Time-to-Collision (TTC) may be appropriate. Traffic conditions (e.g. free flow vs. congested traffic) affect many of the indicators and should be considered when known or obtainable. Self-reported observational data can provide further valuable insights, while being less resource-intensive to obtain, compared to vehicle data analysis. Impact pathways The driving behaviour impact area is not directly influenced by factors addressed under other EU-CEM evaluation areas. However, the technical functioning of the system should be known (see Chapter 4.1.1). Figure 9 shows the output indicators of the driving behaviour impact area. 100 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 9. Output indicators of driving behaviour. Table 6 provides examples of output from driving behaviour evaluation that serve as inputs for other impact areas. Some of these outputs overlap with the recommended indicators above, while others need to be added. Please note that the required input needs must be specified in detail, as they are unique to each project. The table below is not exhaustive. Table 6. Outputs from driving behaviour evaluation required as input for other impact areas. Impact area requiring input Needed input Impact area requiring input User Longitudinal and lateral driving behaviour related to comfort of driving. Frequency of take-over requests. User Traffic safety Longitudinal and lateral driving behaviour, new types of situations (e.g. minimal risk manoeuvres) that did not occur before the introduction of ADS. Traffic safety Traffic flow efficiency Longitudinal and lateral driving behaviour. Traffic flow efficiency Overview of approaches/methods Selecting the right method for evaluating driving behaviour ensures relevant and reliable results. The choice is influenced by the project’s objectives and the resources available to partners. Additionally, the available tools and data shape the feasibility and effectiveness of different approaches (see Chapter 3.3 for further details). Table 7 provides an overview of various approaches used in driving behaviour evaluations. It highlights the strengths, weaknesses, and requirements of each method, helping to identify those that best align with the project’s objectives, resources, and specific evaluation aspects. It is important to note that, even when the cons outnumber the pros, one pro can outweigh multiple cons. 101 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table 7. An overview of approaches and methods for evaluation of driving behaviour. Approach/method Pros Cons Requirements Observing driving behaviour of the ego vehicle (real world)—Objective measurements of the movements of the ego vehicle and the interaction with nearby road users— from the ego vehicle or from fixed locations. True, real-world data representing actual system behaviour in traffic. Limited variety of traffic situations in small field experiments. Extensive data collection required. Time-consuming and costly data processing (especially camera data). Lateral movement adds complexity, like lane line detection, for collecting enough context and for processing. Extensive and highquality data collection and logging capabilities. Access to vehicle data in an agreed format. Segmenting the driving data into distinct driving scenarios (e.g. approaching a vehicle, cut-in, lane change, etc.) allows targeted comparisons between similar situations under different conditions. Driving scenario segmentation allows for more precise, fair comparisons and easier indicator calculation for relevant events. It also makes combining data from different test routes easier. Scenario segmentation can be complicated, especially when analysing sequential scenarios. Precise definitions and thresholds are necessary. Advanced data processing pipeline. Collection of driving scenarios with precise definitions [16]. Observing driving behaviour of the ego vehicle (real world)—Self-reported measurements, e.g. the Driver Behaviour Questionnaire [26]. No need for monitoring equipment. Avoids time-consuming and costly data processing. Allows collection of subjective insights. Entirely subjective data, depends on participant accuracy and honesty. Only certain traffic situations and indicators can be surveyed. Less accurate than quantitative measurements. A well-designed questionnaire or interview protocol. A trained administrator or clear instructions if self-administered. Ideally combined with objective data for validation. Observing situational variables (supplementary activity, part of other methods)—Measured by invehicle or roadside sensors or derived from external databases. For connected automated vehicles, it can be relevant to collect information on incoming messages. Collecting situational variables allow for: - avoiding bias in the results due to different conditions - contextualising the observed vehicle movements - defining proper comparisons (e.g. only Costly to collect and process, especially situational variables. Matching data to external sources can be difficult. Roadside or invehicle data logging capabilities. Accurate synchronisation of timestamps and geolocation. Procedures to match vehicle movement to situational factors, including 102 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility compare under similar weather conditions). Useful for assessing whether the vehicle is driving within or outside its ODD, if that is not logged. (map)matching with external databases. Compatibility between data formats. Observing driving behaviour of the ego vehicle (in a driving simulator)—to investigate how the user influences the vehicle dynamics, including in potentially dangerous situations, and to analyse the transfer of control. Allows studying the impact of the user on the vehicle’s driving behaviour in a controlled environment. Safe testing of rare or dangerous traffic situations that are difficult or unsafe to replicate in field experiments (e.g. if the user takes over control of the vehicle in a critical situation). Repeatable staged scenarios in a controlled environment allow for consistent data collection and comparison. Emotional or cognitive responses of the users may differ from realworld driving, particularly in dangerous situations. A limited number of situations can be tested. Simulator fidelity can limit analysis of highfrequency, fine-grained driver behaviour (e.g. low-level control actions like steering corrections). Learning effects can invalidate repeated tests if not carefully designed. Some indicators equal simulator settings and are thus not meaningful to analyse. A realistic, highfidelity driving simulator (and scenarios) with realistic automated driving functions and user interface. Thorough and validated study design to ensure validity of results. Human factors and vehicle dynamics expertise for interpreting results and adjusting simulator parameters. Traffic scenario simulation Allows for analysis with higher or variable penetration rates, mixed-traffic, and different traffic conditions. Substantial data can be generated quickly. Good supplement to limited field experiments. Costly to develop, calibrate, verify, and validate a model that realistically simulates automated and manual vehicle interaction. Indicators that match simulation settings provide no new insights. For example, setting a 1.6s target headway and analysing the data to find a median target headway of 1.6s. Extensive traffic simulation knowledge. Realistic models for both automated driving and manual drivers (especially if take-over-control processes are simulated). Proper calibration with real-world data. Simulation with Hardwareor Software-in-the-Loop – using a ‘real’ ADS as it would be Realistic usage of the actual ADS in a simulated traffic environment. Challenging to integrate real ADS into the simulation. Team with extensive traffic simulation expertise and 103 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility built into an automated vehicle. Tests how the system performs in a variety of scenarios without risking safety in the real world. May require support from the ADS developer that understands its internal workings. May require large computational power. thorough knowledge of the ADS. Integration of the real ADS with the simulated traffic (e.g. object detection needed). Sufficient computational capacity. Pitfalls and best practices related to different methods Evaluating driving behaviour requires consideration of the methods and approaches used, as these can significantly influence the validity and reliability of the results. Overlooking potential pitfalls may lead to flawed conclusions, misinterpretation of findings, or unintended biases. By proactively addressing the pitfalls, these can be avoided. Table 8 provides some common pitfalls of driving behaviour evaluation and presents actionable best practices to address them. Many of the pitfalls could be avoided if the guidelines in Chapter 3 are carefully followed. Table 8. Pitfalls in the evaluation of driving behaviour and best practices to avoid them. Topic Pitfall Related best practice Measuring driving behaviour Unsuitable logging system. For example, insufficient resolution or frequency, or unreliable logging. Reserve sufficient resources and planning for a suitable logging system. Pilot-test the logging early to confirm data quality. Undecipherable data from CAN-bus, automotive Ethernet, ROS messages, etc. For example, due to proprietary encoding, non-standard implementation, complex data structure, and safety measures like encryption or obfuscation. Agree on the data format, content, and sharing protocol from test site to evaluation, in line with recommendations in Chapter 3.4. Reserve sufficient budget and expertise for data analysis. The logging system is not piloted, leading to late discovery of poor data quality. Develop and follow a good piloting plan. Adapt the project timeline if necessary to allow for pilot tests. Unclear whether automated mode was active, or which automated driving functions were in use. Log automation mode status explicitly, including transition of control: e.g. take-over requested, take-over completed, initiated by driver, at ODD border, within ODD, and so on. Verify correctness of the logging during piloting. Incompatible test routes with substantial differences in conditions and characteristics. Collect baseline and treatment data from all test routes under similar conditions. 104 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility This makes combining data from all test routes difficult and comparison of calculated indicators misleading. Segment the driving data into distinct, welldefined driving scenarios with accurate contextual metadata. Calculate indicators for the driving scenarios and find similar scenarios across the combined data. Compare the treatment and baseline indicators from similar scenarios. They should reflect the driving behaviour rather than the conditions around the vehicle. Comparing data of different lengths. For example, one car-following scenario can be very long, another very short, leading to e.g. incomparable variances. Standardise scenario lengths. Split long scenarios into multiple ones of standardised lengths. For example, car-following scenarios of 40 s are split into two 20 s ones. Analysing video data Unclear video data annotation process, no video annotation tools, or insufficient resources for annotation. Specify annotation requirements in advance. That is, what should be labelled, how, and why, per research question. Note what kind of video is required (internal, external, viewing angles, frame rates). Reserve sufficient resources for planning and performing the annotation work, including potential lane marking detection. Adhere to privacy requirements. For example, blur faces or licence plates and note restrictions for filming in sensitive areas. Ensure that appropriate software tools for video annotation are available and compatible with required input and output formats. Traffic scenario simulation Inaccurate vehicle models (both humanand ADS-driven) Calibrate, verify, and validate models using realworld data. Use data from pilots and/or existing datasets. Collaborate with ADS developers to configure parameters that cannot easily be calibrated. Inaccurate or unrealistic driver, passenger or pedestrian models Calibrate, verify, and validate models using realworld data. Supplement with literature for bestfit parameters. Insufficient time and resources for calibration and validation. Include calibration and validation in the work plan with a dedicated budget and milestones. No data available for calibration and validation. Plan reference data acquisition to be included in the agreement of data that the project will provide for evaluation. Formalise data sharing agreements early (see Chapter 3.4). 105 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4.2. Human 4.2.1. User 4.2.1.1. Introduction Definition and scope EU-CEM defines the user evaluation area as follows: User evaluation addresses the interaction of people with the CCAM system: their opinions, expectations, and awareness of the technology, and how they use it. Interaction with the CCAM system is understood broadly here. Users can be drivers interacting directly with the CCAM system, for example when taking over control of the vehicle when the ODD conditions are no longer met. Users can be passengers interacting with a CCAM system, for example via an app or screen, or even by just sitting in the vehicle. Users may also be receivers or senders of goods being delivered by CCAM systems. User evaluation may also address "non-users", namely people who are affected by CCAM but do not directly use it; for example, other road users (both motorised and non-motorised) and residents of an area where CCAM operates. User evaluation provides important input to many impact areas, such as people mobility and traffic safety. User evaluation results can also be used for further development of CCAM systems or for developing business cases and marketing. The user evaluation chapter has the following scope: • User behaviour with a CCAM system, for example take-over control situations (reaction time, takeover time, changes in driving behaviour like sudden deviation in speed or position in lane), use of time in the vehicle (non-driving-related activities), observation or measurements of interaction via interfaces and in-vehicle systems, and observation or measurements of the effects on behaviour of "non-users". • Experience, for example the user experience (pleasant, easy, useful, etc.) and the physical and mental state during the drive or ride (comfort, convenience, stress, motion sickness, feelings of safety and security, concerns about cybersecurity, data protection and privacy, etc.), trust in the system, suitability of the system for specific target groups (disability, age groups, etc.), and feelings and experiences of "non-users" confronted with CCAM. • Expectations, for example, on willingness to use the CCAM system, concerns about societal impacts and consequences for users and non-users, usefulness for individuals in their envisaged future situations, or usefulness for specific groups. User evaluation in EU-CEM does not include: • Technical validation of user interfaces (see Chapter 4.1.1) or usability testing. • Financial implications or business models—See Chapter 4.3.1. • Experiences or opinions of service providers, managers, operators, and other stakeholders—See Chapter 4.3.1. • Equity impacts for different user groups—See Chapter 4.4.5. Background The three components of the user evaluation (Figure 10) have a rich and diverse theoretical background. Behaviour, experience, and expectations of users are studied in many disciplines like behavioural science, social science, economics, human factors, engineering, marketing, and so on. Each of them has their own focus, 112 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility • Personal situation in terms of financial means, (dis)ability, employment, family, etc. • Residential setting and distance to activities, transport infrastructure • Trip purpose, such as commute, leisure or shopping • Availability of travel modes, vehicle ownership • Habits, motivations, and attitudes toward travel modes • Familiarity with travel modes, availability of information about travel modes • (Perceived) quality, comfort, ease of use, safety and personal security of available modes • Costs of available travel modes • Weather • Real-time traffic information, such as congested routes CCAM may affect several of the above factors and their relevance to an individual’s travel choices. For example, if the generalised travel costs of personal cars drop, people may travel more by car. More flexible mobility services may lead to combining trips previously done separately. Better access to different locations may also affect the activities that people engage in, and new services can change the need for certain types of trips. For example, robot deliveries can replace trips to shops or restaurants. The most important components of the people mobility impact area are summarised in Figure 12. Figure 12. Main components of the people mobility impact area. People mobility impacts at the level of an individual traveller aggregate to impacts on the transport system level (see Chapter 4.3) in terms of modal split and amount of travel on the network over a longer period. They can have direct impacts on other areas as well, such as quality of life (Chapter 4.2.3), traffic safety (Chapter 4.3.4), energy and environment (Chapter 4.3.6), and traffic flow efficiency (Chapter 4.3.5). In the long term, changes in generalised travel costs influence the accessibility (Chapter 4.3.7) of different activities and their locations. This is can lead to changes in land use (Chapter 4.4.1) and the liveability of areas (Chapter 4.4.2). CCAM may not only 113 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility impact its users but can also influence the travel behaviour of non-users if, for example, increased car usage reduces the willingness to walk or cycle due to safety concerns. Whether increasing people mobility is a societal goal depends on whether increased people mobility aligns with other outcomes that the society may have prioritised, such as economic vitality, improved environmental conditions, and equity. These objectives relate to questions such as whether CCAM will generate more (motorised) traffic or increase travel costs. The impacts of CCAM must be evaluated within the broader context of other trends in transport and mobility. 4.2.2.2. Guidelines Indicator recommendations Indicators recommended for evaluation in every project addressing people mobility impacts are listed in Table 13 below. The indicators describe individual choices and behaviours. Impacts are measured as differences or changes between the baseline and treatment scenarios for these indicators. Impacts can be reported as absolute values (with the unit below) or relative value (%). If certain indicators cannot be evaluated, the reason must be documented following the ‘comply or explain’ principle of EU-CEM. The listed indicators are not exhaustive; in addition to these, each project should define its own indicators in accordance with Guideline 3.12. Table 13. Indicators recommended to be evaluated by every project addressing people mobility. Indicator short name Definition Unit Mode share Share of trips of individuals by travel mode, potentially per trip purpose. % Amount of travel Number of trips or kilometres or hours an individual travels in total within a specific time, like one year. Number, kilometres, hours, minutes Length of trips Average distance or duration of trips. Kilometres, hours, minutes Timing of trips Hourly distribution of trips (starting time). Number of trips performed at defined time ranges Value of travel time Perceived costs of travel time; including accepted additional travel time with a CCAM system, change in the monetarised value of travel time. €/time unit Impact pathways Impact pathways illustrate how changes caused by CCAM, and assessed in other impact areas, affect the outcomes of this impact area. Mapping these cause-and-effect relationships helps to scope the evaluation by grounding it in established theories or empirical findings. It clarifies how localised effects can escalate to larger scale impacts, shows whether one area influences another, and reveals how these changes ripple through vehicles, individuals, the transport system and beyond. Using these pathways can help identify where additional information is needed, where collaboration with other impact areas is required, or where certain aspects fall outside the scope of the assessment. People mobility is influenced by impacts on users (Chapter 4.2.1), services and operation (Chapter 4.3.1) and economic activity and employment (Chapter 4.4.3). This is depicted in the impact pathway below (Figure 13). 114 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The figure offers a general overview that can be adapted or extended to specific CCAM systems studied. Each project should tailor it by specifying relevant indicators and units. Figure 13. Input needed from the other impact areas for people mobility impact assessment and its outputs. Table 14 provides examples of people mobility impact assessment that serve as inputs for other impact areas. Some of these outputs overlap with the recommended indicators above, while others need to be added. Please note that the required input needs must be specified in detail, as they are unique to each project. The table below is not exhaustive. Table 14. Outputs from people mobility impact assessment required as input for other impact areas. Impact area requiring input Needed input Quality of life Amount of active and sedentary travel Services and operation Travel behaviour and needs Transport activity and fleet composition Travel patterns, vehicle ownership Traffic safety Travel patterns Traffic flow efficiency (macroscopic models) Value of travel time Accessibility Destinations 115 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Overview of approaches and methods Selecting the right method for evaluating people mobility ensures relevant and reliable results. The choice is influenced by the project’s objectives and the resources available to the partners. Additionally, the available tools and data shape the feasibility and effectiveness of different approaches (see Chapter 3.3 for further details). Table 15 provides an overview of various approaches used in people mobility evaluations. It highlights the strengths, weaknesses and requirements of each method, helping to identify those that best align with the project’s objectives, resources, and specific evaluation aspects. It is important to note that, even when the cons outnumber the pros, one pro can outweigh multiple cons. Table 15. An overview of approaches and methods for people mobility impact assessment. Approach/method Pros Cons Requirements Subjective methods (questionnaires, interview, focus groups, etc.) Captures people’s subjective ideas, feelings, etc. Relatively easy and costefficient. Flexible. Existing or standard questionnaires enable consistency. Subject to response biases: subjective bias, non-response, social desirability bias, interviewer bias, etc. Quality of data depends on participant honesty and recall. Survey platform or tools (online or paper). Trained staff for designing and moderating focus groups. Stated preference survey and choice modelling Can determine willingness to pay or use CCAM systems by characteristics. Bias in answering and lack of representativeness. Stated preference answers may not provide reliable descriptions of how mobility-related habits would change. Potentially costly to recruit a representative participant group for data collection. Large-scale participation from a representative population for reliable results. Expertise in design of stated preference experiment. Suitable expertise and software for modelling the data. Accurate description of CCAM system needed. Modelling— Activity-based models Can run individual-triplevel simulations. Captures complex behaviours. Dataand setup-intensive. Potentially long simulation run times. Model requires calibration to ensure realism. Model needs to be developed or available and suitable for use. Sufficient data for calibration. Expertise in network simulation. Modelling— System dynamics Handles scenarios representing long time periods (months or years). Aggregated population approach: less detail on individuals. Needs clear boundaries for model scope. Established system dynamics tool. Model needs to be available and suitable for use. 116 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Once set up, quick scenario runs are possible. Includes feedback loops. Requires historic data for calibration. Dataand time-intensive to set up. Historic data for calibration. Expertise in system dynamics modelling. Travel surveys, revealed preferences data, census data and other historical external data for understanding current mobility patterns (baseline) Established methods. Long-term historic data. Often representative at a population level. Typically populationaggregated; hard to obtain highly detailed data. Typically a "snapshot" and may be outdated. Agreements for data provision. Carefully planned baseline for the societal scenario. Data privacy plan. New and emerging data forms, such as mobile phone or social media tracking, for understanding current mobility patterns (baseline) Can develop a contemporary and dynamic baseline of real-world behaviour. Can gather data on individuals’ behaviour and preferences. Limited representativeness. May lack personal details, like characteristics or trip purpose. Possible data privacy issues. Difficult to acquire access. Agreement on data provision. Create synthetic populations alongside other data, such as census data. Data privacy plan. Gamification, serious games Can produce rich data without exposure to real CCAM. Interactive scenarios. Development can require substantial resources. Game environment may not capture real-world constraints (time, cost, risk). Developers for game design. Experienced facilitators. Pilot testing to confirm realism and data validity. Pitfalls and best practices Evaluating people mobility impacts requires consideration of the methods and approaches used, as these can significantly influence the validity and reliability of the results. Overlooking potential pitfalls may lead to flawed conclusions, misinterpretation of findings or unintended biases. By proactively addressing the pitfalls, these can be avoided. Table 16 provides some common pitfalls of people mobility impact assessment and presents actionable best practices to address them. Table 16. Pitfalls in people mobility impact assessment and best practices to avoid them. Topic Pitfall Related best practice Future mobility options People may struggle to imagine new options or accurately assess their future behaviour. Aim to make the user experience as immersive as possible. Consider virtual reality and immersive experiences. Use stories and scenarios to make abstract ideas more concrete. Ask specific questions rather than 'what ifs'. 117 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Subjective scales Variations in how participants interpret scales, especially in large, multicultural or multilingual surveys. Include guidelines or examples to anchor responses. Use standardised surveys. Use cross-cultural adaptation methods. CCAM impact Failing to distinguish between impact of the CCAM system being tested and other concurrent system impacts (that may occur without CCAM). Define a societal scenario that clearly distinguishes what changes with the introduction of CCAM and what other transport developments may take place. If relevant, identify changes in the environment or socio-economic situation, or envisaged changes in personal life such as retirement. Habits and cultural norms Established habits and cultural norms may remain unchanged or shift slowly, while future generations could differ from current ones. Recognise limitations and applicability of generalisations from specific population/socio-demographics of study participants. Recruiting representative user groups Difficulty in recruiting a representative user group relevant to the target population. Recruitment may skew towards people with interest in the technology and time available. Adjust target population to characteristics of recruited respondents. Develop synthetic populations. 4.2.3. Quality of life 4.2.3.1. Introduction Definition and scope EU-CEM defines the quality-of-life impact area as follows: The quality-of-life impact area addresses the holistic effects of the CCAM system on individuals' physical, mental, social, and financial well-being. The quality-of-life impact area addresses the holistic effects of CCAM on individuals' physical, mental, social, and financial well-being. In the assessment, both objective and subjective perspectives are considered and interpreted relative to an individual’s values and goals [30]. This chapter considers quality of life and describes how the concept can be assessed in the CCAM context. The well-being of communities is addressed in Chapter 4.4.2 (Liveability). Background Quality of life describes “individuals’ perception of their position in life in the context of culture and value systems in which they live and in relation to their goals, expectations, standards, and concerns” [31]. Quality of life is a multidimensional construct, and its measurement is complex. It encompasses both subjectively experienced satisfaction and objectively measurable achievements [32]. A simple sum of positive and negative factors does not adequately capture quality of life, as individuals may experience identical conditions differently depending on their perception of what they could achieve in life [33]. 118 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility The concept of quality of life emerged to address the shortcomings of more simple measures. For example, in the medical literature, quality of life emphasises the fact that just extending the length of life might not be a meaningful goal if the remaining life is of poor quality [34]. Within the transport domain, it could be similarly argued that minimising fatalities and injuries sustained in traffic is not enough for ensuring a good level of quality of life. For example, replacing a zebra crossing at a busy road with a bridge may prevent accidents between cars and vulnerable roads users, but at the same it may reduce convenience and accessibility, if using the bridge means additional climbing or taking a detour. This means that such a traffic safety measure may strengthen the community severance effect of the road. Community severance means that a transport infrastructure creates a physical or psychological barrier between two areas, which impairs the well-being of people moving between them [35][36]. Quality of life can be studied in terms of individuals' physical, mental, social, and financial well-being. Physical well-being refers to fitness, or the ability to perform everyday tasks in daily life, and years lived without illness or disabilities. Mental well-being refers to positively experienced mental states and the perception of being able to reach goals, whilst having positive expectations towards life and experiencing a good standard of life. Social well-being means having meaningful and supportive relationships and feeling a sense of belonging to groups. Finally, financial well-being means having sufficient financial assets or purchasing power to meet personal needs, and being able to earn more assets, for example by working. The main components of the quality-of-life impact area are summarised in Figure 14. Figure 14. Main components of the quality-of-life impact area. The four dimensions of well-being (physical, mental, social, financial) are influenced by accessibility or access to needed locations, the built environment, and traffic [30]. Therefore, CCAM systems may affect all these wellbeing dimensions. For example, automated cars could improve access of individuals to specific locations and thus improve their economic, social, and mental well-being. On the other hand, increased car usage might lead to more sedentary travel and a reduction in active travel [37]. As active travel is an important source of physical activity [38], CCAM systems could thus decrease the physical well-being and possibly also mental well-being of 119 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility its users. Automated driving could also change what activities can be performed while travelling [39] (see Chapter 4.2.1). This can make multitasking while travelling easier than before. The ability to engage in activities during travel may increase flexibility but may also induce stress by creating an expectation to use travel time productively. This may reduce the ‘positive utilities’ of travel time, such as relaxation or time off [40]. This can have complex influences on mental, social, and financial well-being. CCAM systems could also change the amount of time people spend travelling (see Chapter 4.2.2). Some people might perceive the travel time as enjoyable or relaxing on its own [41]. Therefore, minimising travel time may not be an important objective for everyone [42]. However, excessive travel time has a negative effect on the subjective experience of commuting and overall well-being [41]. Satisfaction with life is a related (but narrower) concept than quality of life. It focuses on subjective experiences of well-being [43], which are positively correlated with objectively measurable factors recognised as relevant to well-being. For example, severe injuries from crashes often reduce both subjectively experienced and objectively measured well-being. Thus, the subjective experience of well-being can reflect tangible, objectively measurable factors. Consequently, satisfaction with life or travel could be used as a proxy for quality of life. Subjective experiences while travelling, access to activities, and activities performed while travelling can influence subjective well-being [44]. Satisfaction with travel can be relevant for the acceptance of CCAM systems. Low satisfaction with the current mode of travel is linked to a greater willingness to use automated vehicles [45]. 4.2.3.2. Guidelines Indicator recommendations EU-CEM does not currently provide recommendations for quality-of-life indicators that should be used by all projects. It is recommended that the projects measure quality of life with scores calculated based on published, previously validated questionnaires and scales and that they are explicit in how the scores have been calculated. Standardised scores are useful when comparing quality of life between groups of people, for example when assessing the equity impacts (Chapter 4.4.5). Projects could also conduct qualitative assessment of the qualityof-life dimensions, such as expected changes in physical, mental, social and financial well-being. Impact pathways Impact pathways illustrate how changes caused by CCAM, and assessed in other impact areas, affect the outcomes of this impact area. Mapping these cause-and-effect relationships helps scope the evaluation by grounding it in established theories or empirical findings. It clarifies how localised effects can escalate to larger scale impacts, shows whether one area influences another, and reveals how these changes ripple through vehicles, individuals, the transport system and beyond. Using these pathways can help identify where additional information is needed or where collaboration is required with other impact areas, or to ascertain whether certain aspects fall outside the scope of the assessment. Quality of life is influenced by impacts on people mobility (Chapter 4.2.3), accessibility (Chapter 4.3.7), liveability (Chapter 4.4.2), and traffic safety (Chapter 4.3.4). This is depicted in the impact pathways below in Figure 15. The figure offers a general overview that can be adapted or extended to specific CCAM systems studied. Each project should tailor it by specifying relevant indicators and units. 120 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 15. Inputs needed from the other impact areas for the quality-of-life impact assessment. Table 17 provides examples of quality-of-life impact assessment that serve as inputs for other impact areas require. Some of these outputs overlap with the recommended indicators above, while others need to be added. Please note that the required input needs must be specified in detail, as they are unique to each project. The table below is not exhaustive. Table 17. Outputs from quality-of-life impact assessment required as input for other impact areas. Impact area requiring input Needed input Liveability Well-being of individuals Equity Distribution of quality-of-life impacts across different groups of people Overview of approaches and methods Selecting the right method for evaluating quality of life ensures relevant and reliable results. The choice is influenced by the project’s objectives and the resources available to the partners. Additionally, the available tools and data shape the feasibility and effectiveness of different approaches (see Chapter 3.3 for further details). Table 18 provides an overview of various approaches used in quality-of-life evaluations. It highlights the strengths, weaknesses, and requirements of each method, helping to identify those that best align with the project’s objectives, resources, and specific evaluation aspects. It is important to note that, even when the cons outnumber the pros, one pro can outweigh multiple cons. 121 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table 18. An overview of approaches and methods for evaluation of quality of life. Approach/method Pros Cons Requirements Focus groups and interviews Allows for consideration of the multidimensionality and complexity of quality of life. Participants may not have experience with the CCAM system. Only subjective results. Participants should be provided a good experience or description of the CCAM system. Careful selection of participants. Surveys Previously validated survey instruments are available in the literature [46][47][48][49]. Validated questionnaire available. Participants may not have experience with the CCAM system. Only subjective results. Participants should be provided a good experience or description of the CCAM system. Expert assessment of the impact pathways to quality of life Efficient to implement. Can cover a wide range of impacts. Limited to the impact pathways recognised by experts. Only subjective results. Assessment of the relevant areas with impact pathways to quality of life. Multidisciplinary team of experts. Pitfalls and best practices Evaluating quality of life impacts requires consideration of the methods and approaches used, as these can significantly influence the validity and reliability of the results. Overlooking potential pitfalls may lead to flawed conclusions, misinterpretation of findings, or unintended biases. By proactively addressing the pitfalls, these can be avoided. Table 19 provides one common pitfall of quality-of-life evaluation and presents actionable best practices to address them. Table 19. Pitfalls in evaluation of quality of life and best practices to avoid them. Topic Pitfall Related best practice Limited viewpoint Evaluation concerns only a few related impact areas, like traffic safety and people mobility, not enabling holistic assessment of the quality of life. Quality of life is a broad concept that should not be evaluated from narrow perspectives. If a comprehensive evaluation is not feasible, it is more appropriate to discuss the potential impact on quality of life through the impact pathways rather than claiming to evaluate it. 128 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility especially if some user groups are underrepresented. user groups properly represented. Use of incentives or targeted outreach. Pitfalls and best practices Evaluating impacts on services and operation requires consideration of the methods and approaches used, as these can significantly influence the validity and reliability of the results. Overlooking potential pitfalls may lead to flawed conclusions, misinterpretation of findings, or unintended biases. By proactively addressing the pitfalls, these can be avoided. Table 23 provides some common pitfalls of services and operation evaluation and presents actionable best practices to address them. Table 23. Pitfalls in evaluation of services and operation and best practices to avoid them. Topic Pitfall Related best practice Traffic management in mixed traffic Changes in traffic management due to CCAM can affect other road users as well. A solution that is best for CCAM might be suboptimal for the overall network. Decisions on traffic management may vary between operators. Evaluate both direct and indirect impacts on all road users across relevant impact areas, such as traffic flow efficiency and safety. Conduct sensitivity analyses to evaluate how different traffic management strategies influence outcomes. Carefully define the societal scenario and the scope of evaluation. Confidentiality of business models Stakeholders may be unwilling to share details of their business models for evaluation. Carefully frame research questions to avoid confidentiality issues while still addressing the evaluation goals. Establish special partnerships between service providers and evaluators to maintain confidentiality while sharing valuable insights. Insufficient stakeholder engagement Evaluation results may lack depth or relevance if key stakeholders are not adequately involved. Identify all relevant stakeholders early. Facilitate early and active involvement of all relevant stakeholders. 129 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4.3.2. Logistics 4.3.2.1. Introduction Definition and scope EU-CEM defines the logistics impact area as follows: The logistics impact area addresses how CCAM affects the efficiency, effectiveness, and adaptability of the logistics process. It examines how CCAM can change the freight transport, and material flows in supply chain processes. The logistics impact area examines how CCAM can affect goods transport in different environments, including on public roads with mixed traffic and in confined areas such as ports and terminals. CCAM has the potential to improve the reliability of operations and increase their efficiency, for example by optimising energy use and allowing operations during off-peak hours. In addition, worker safety could be enhanced. In this chapter, logistics covers at least the following use contexts: • Operating environments o Motorway driving in mixed traffic, for example auto-follower, platooning, entering and exiting motorways, lane changing, managing lane closures, and navigating roadworks. o Driving in confined areas, for example ports, terminals, industrial areas, construction sites, and mining or quarry sites, which typically have limited and controlled access. Mixed traffic and the presence of vulnerable road users may be included. • Functions o Gate Access, for example procedures for planning, approaching, parking, cargo unit identification, waiting, and managing entry and exit to confined areas. o Inspections by customs and other authorities, for example international border crossings, customs inspections, and other road controls (e.g. weight checks, goods inspections, smuggling control) and handling at international ports and terminals. o Parking and movements of truck-and-trailer combinations at slow speeds in narrow or confined environments. o Charging, for example automated charging processes for truck and trailers at both public and private locations. o Automated loading and unloading of containers and cargo, including connecting and disconnecting trailers. o Remote operation, for example remote monitoring, take-over, and control by fleet managers, transport managers or authorities. • Logistics service concepts o First-and-last-mile logistics in urban, regional or rural transport using all types of road vehicles, including public service vehicles, such as waste collection and recycling. o Hub-to-hub logistics, covering both shortand long-distance operations on dedicated, restricted or public roads, often requiring permits. o Intermodal and transshipment, for example transfer of goods between different transport modes, particularly in ports and terminals. o Logistics using new innovative concepts with CCAM, for example low-speed transport using small vehicles and load units, 24/7 operations, and implications for warehouses, logistics hubs, and so on. 130 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Across different use contexts, the impacts of CCAM on continuity in logistics operations should be considered. New situations potentially affecting continuity, such as changes in the ODD requirements of CCAM systems or system disruptions, may arise. Several other impact areas address viewpoints complementary to this chapter. For example, impacts of CCAM on logistics services and their operation are addressed in Chapter 4.3.1 on services and operation, and the economic aspects in Chapter 4.4.3 on economic activity and employment. The environmental efficiency of transportation is addressed under environment and energy in Chapter 4.3.5. Background Logistics concerns “the efficient transfer of goods from the source of supply through the place of manufacture to the point of consumption in a cost-effective way while providing an acceptable service to the customer” [53]. Thus, it deals with the movement and management of materials, products, and resources in supply chains, from production to delivery to end customers. Effective logistics requires balancing supply and demand, managing inventories, and ensuring consumer satisfaction, whether the customer is another company or an endconsumer. The goal is to deliver the correct products on time, add value for customers, and minimise costs for all parties involved. Cost-effectiveness and customer service need to be balanced effectively. Logistics is related to road transport, as a substantial part of goods is transported as road freight. Relevant considerations include vehicle and delivery operation choices, load planning, and route selection. Logistics vehicles such as trucks can require significant long-term investments. Logistics companies strive for efficient use of assets by utilising vehicles that are fit for purpose and optimising utilisation rates while minimising both fixed and variable costs. Logistics encompasses different ways of moving goods, depending on the transported volume and type of goods, costs of transport, and transit speed. It includes (a chain of) physical flows between ports, terminals, hubs, warehouses, and distribution centres. Physical flows are supported by the flow of information, which is especially important in the context of automation and digitalisation. CCAM impacts on logistics can be studied for entire logistics chains, or for their separate parts, for example logistics operations on public roads, in warehouses, terminals, ports, customs or interchanges (Figure 18). 131 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 18. Main components of the logistics impact assessment. A transport chain typically includes multiple phases, each encompassing specific tasks [54]: • Composition: Assembling loads on pallets, containers or tankers, depending on the type of cargo. • Modality and inter-modality: Moving cargo along the transport chain using one or more transport modes. Intermodal transport includes terminals where trailers or containers are transhipped onto another transport mode, such as rail, ship or air. Rail terminals, seaports and airports serve as key facilities for these transfers. • Road transport: Conducting line haul operations between depots or performing multi-stop trips originating and concluding at a depot. • Border crossing: Managing the movement of cargo into another country, including customs inspections. • Last mile: Refers to the final stage of the transport chain involving the distribution of goods to their final destinations, such as stores, consumers’ homes or pick-up locations. This segment has grown rapidly due to the rise in online sales. With the shift toward a circular economy, first-mile logistics activities, such as pick-ups at the beginning of the transport chain, have also gained increasing importance. All phases of the transport chain include uncertainties related to transported volumes, delays, quality, demand, and access to information. One substantial challenge in logistics management is the complexity of the supply network, which involves multiple actors in different roles and various types of activities. Additionally, road transportation is continuously exposed to traffic and weather conditions that affect delivery times and routing, causing delays. Workforce shortages, especially among drivers, and a disparity between required and available skills are affecting road transport capacity and efficiency. A consequence may be delays in deliveries, inefficient vehicle use, and high fluctuations in transport costs. Additionally, vehicles often remain idle due to mandatory driver rest periods. CCAM has the potential to address workforce shortages and mitigate inefficiencies associated with rest times. 132 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Last-mile deliveries and first-mile pick-ups have distinct challenges, such as low delivery volumes, tight time schedules, and high costs per delivery. CCAM could offer new and more cost-efficient options for addressing the challenges of lastand first-mile deliveries. 4.3.2.2. Guidelines Indicator recommendations The result indicators recommended to be evaluated by every project addressing logistics are listed in Table 24 below. Impacts are measured as differences/changes between the baseline and treatment scenarios for these indicators. Impacts can be reported as absolute values (with unit below) or relative values (%). If it is not possible to evaluate some of the indicators below, the reason for this should be reported as part of the ‘comply or explain’ principle of the EU-CEM. The indicators listed below are not exhaustive; in addition to these, each project should define indicators of their own in line with Guideline 3.12. Table 24. Indicators recommended to be evaluated by every project addressing logistics. Indicator short name Definition Unit Transport efficiency Tonne-km transported per vehicle-kilometre driven Tonne-km/vehiclekm Duration Sum of transshipment time, waiting time, and driving time Hours Punctuality Deviation from the targeted delivery time % and time Transshipment time Change in transshipment time at terminals, ports, customs, interchanges, etc. % and time Waiting time Change in waiting time at terminals, in road transport, etc. % and time Impact pathways Impact pathways illustrate how changes caused by CCAM, and assessed in other impact areas, affect the outcomes of this impact area. Mapping these cause-and-effect relationships helps scope the evaluation by grounding it in established theories or empirical findings. It clarifies how localised effects can escalate to larger scale impacts, shows whether one area influences another, and reveals how these changes ripple through vehicles, individuals, the transport system and beyond. Using these pathways can help identify where additional information is needed or where collaboration is required with other impact areas, or to ascertain whether certain aspects fall outside the scope of the assessment. Logistics is influenced by changes in operations, such as driving speed and accommodation of speed to the next logistics process, and use of infrastructure during off-peak hours. These are addressed under services and operation (Chapter 4.3.1). In addition, road transport is affected by traffic flow efficiency (Chapter 4.3.5). The demand for goods transport is affected by economic activity and employment (Chapter 4.4.3). These linkages are depicted in the impact pathway in Figure 19. The figure offers a general overview that can be adapted or extended to the specific CCAM systems studied. Each project should tailor it by specifying relevant indicators and units. 133 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 19. Input needed from the other impact areas for logistics impact assessment and its outputs. Table 25 provides examples of outputs of logistics impact assessment outputs that serve as inputs for other impact areas. Some of them overlap with the recommended indicators above, while others need to be added. Please note that the required input needs must be specified in detail, as they are unique to each project. The table below is not exhaustive. Table 25. Outputs from logistics impact assessment required as input for other impact areas. Impact area requiring input Needed input Services and operation Traffic safety Energy and environment Transport patterns Transport activity and fleet composition Transport patterns, vehicle fleet Overview of approaches and methods Selecting the right method for evaluating logistics ensures relevant and reliable results. The choice is influenced by the project’s objectives and the resources available to the partners. Additionally, the available tools and data shape the feasibility and effectiveness of different approaches (see Chapter 3.3 for further details). Table 26 provides an overview of various approaches used in logistics evaluations. It highlights the strengths, weaknesses, and requirements of each method, helping to identify those that best align with the project’s objectives, resources, and specific evaluation aspects. It is important to note that, even when the cons outnumber the pros, one pro can outweigh multiple cons. 134 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table 26. An overview of approaches and methods for evaluation of logistics. Approach/method Pros Cons Requirements Field experiment on public roads Involves real stakeholders and operations, yielding genuine, real-world data. Allows calibration and validation of simulations models. Conducting a test in real traffic is difficult, time-consuming, and expensive. The safety of participants and other road users requires careful planning. Equipped vehicles. Access to logistics areas and relevant road network. Permits to perform tests and collect data. Test track More controlled environment than on public roads. The safety of participants and other road users is easier to ensure. More realistic than a simulator. Simplified environment. Less natural traffic flow. Test effect leading to participants being more alert, traffic rule compliant, or trusting the system more than in the real world. May still be difficult, costly, and time-intensive to arrange. Equipped vehicles. A suitable test track with a layout to match the tested logistics operation. Budget for track rental, staffing, and setup. Simulation of logistics operations Systematically explores multiple scenarios with minimal risk. Low-cost iteration compared to field experiments. The simulation depends on model accuracy. Model accuracy depends on correct assumptions about vehicles, the environment, and behaviours. A realistic model of the logistics operation and operational domain. Accurate data for calibration. Simulation expertise. Pitfalls and best practices Evaluating impacts on logistics requires consideration of the methods and approaches used, as these can significantly influence the validity and reliability of the results. Overlooking potential pitfalls may lead to flawed conclusions, misinterpretation of findings or unintended biases. By proactively addressing the pitfalls, these can be avoided. Table 27 provides some common pitfalls of logistics evaluation and presents actionable best practices to address them. Table 27. Pitfalls in evaluation of logistics and best practices to avoid them. Topic Pitfall Related best practice Simulations of logistics operations Different models provide contradictory results due to inconsistent assumptions or incomplete datasets. Validate and calibrate models with real-world data. Involve domain experts to check the realism of assumptions. 135 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Test track The test environment does not fully cover all the important aspects of the logistics operation under evaluation. Identify different alternative test environments and locations suitable for the operation under evaluation. If necessary, split the testing across different tracks to evaluate different aspects or parts of the logistics operations. Document any limitations due to controlled settings and consider additional validation if needed for realism. Field experiment on public roads The real world is complex, multifold, and unpredictable, making it challenging to capture all factors that affect logistics operations. Lack of permits and support from authorities and needed stakeholders. Identify key corridors and demonstration sites where local stakeholders are invested in CCAM or logistics development. Collaborate with living-labs with an existing network of partners and regulatory pathways. Leverage these collaborations to ensure the results have real impact for users, stakeholders, businesses, and society. Allow sufficient time for permit applications and stakeholder agreements. 4.3.3. Transport activity and fleet composition 4.3.3.1. Introduction Definition and scope EU-CEM defines the transport activity and fleet composition impact area as follows: The transport activity and fleet composition impact area addresses the impacts of CCAM on the total amount of realised travel and transport as well as on the number and types of vehicles used. Transport activity refers to the total amount of realised travel on a road network over a certain period. It can be expressed in terms of vehicle kilometres travelled (VKT) (also called mileage or kilometrage), passenger kilometres travelled (PKT), or tonne-kilometres travelled (TKT). It results from the movements of people (People mobility, Chapter 4.2.2) and commercial transport of goods (Logistics, Chapter 4.3.2) subject to the constraints posed by service supply (Services and operation, Chapter 4.3.1) and network conditions (Traffic flow efficiency, Chapter 4.3.5). Transport activity covers potential effects of CCAM on the total VKT, PKT or TKT. This includes changes in modal split and spatial and temporal distributions both within and outside the operational design domain (ODD) of the CCAM system. Vehicle fleet composition focuses on how CCAM influences the total number and characteristics of the vehicles in the transport system. Background Van Wee [55] defines four main characteristics of the transport system: 1. The transport volume or total transport activity 2. The composition of traffic and transport in terms of modal split and vehicle categories 3. The spatial division of transport activity per vehicle category 136 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4. The temporal division of transport activity per vehicle category. Spatial division refers to how traffic is distributed across the road network, whereas temporal division describes when travel occurs, such as the time of day, week or month. CCAM can affect these characteristics through several mechanisms, depending on which systems are introduced, how they operate, and how they are taken into use. The resulting changes in transport activity, whether an increase or a decrease in total VKT or shift between travel modes, can vary widely: • People mobility (Chapter 4.2.2): Automation of passenger cars may cause travellers to use cars more often or over longer distances, leading to higher car VKT. Conversely, automated public transport could draw users away from personal cars, decreasing overall VKT. • Logistics (Chapter 4.3.2): Shifting from larger vehicles to multiple smaller delivery robots can increase VKT, but the additional kilometres could be covered by smaller, electric vehicles. CCAM may also change the spatial and temporal distribution of VKT. For example, limited operational design domains might lead to automated car users preferring motorways where automation can be used, even if travel times increase. Smaller delivery robots could shift freight traffic to new routes or to operate at different times of day compared to trucks and vans. The possibility of automated truck platoons driving at night or working while travelling could affect trip schedules. Transport activity and fleet composition impacts have important consequences for other impact areas. For example, the overall impact of CCAM on energy use and emissions (Chapter 4.3.6) heavily depends on how VKT changes per mode. Automated personal cars may drive more efficiently than human-driven vehicles, but any gains could be offset if the total car VKT increase sufficiently to raise total emissions and energy consumption. Figure 20 shows the main components of the transport activity and fleet composition impact area. Figure 20. Main components of the transport activity and fleet composition impact area. 137 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4.3.3.2. Guidelines Indicator recommendations The result indicators recommended to be evaluated by each project addressing transport activity and fleet composition are listed in Table 28 below. Impacts are differences/changes between the baseline and treatment scenarios for these indicators. An impact can be reported as an absolute value (with unit below) or relative value (%). If it is not possible to evaluate some of the indicators below, the reason for this should be reported as part of the ‘comply or explain’ principle of the EU-CEM. The indicators listed below are not exhaustive; in addition to these, each project should define indicators of their own in accordance with Guideline 3.12. Table 28. Indicators recommended to be evaluated by each project addressing transport activity and fleet composition. Indicator short name Definition Unit Total kilometres travelled The total kilometres travelled in a defined road network over a specified period of time, per travel mode and specified units of time and space within and outside of the operational design domain. VKT, PKT, and/or TKT Distribution of kilometres travelled over time The distribution of total kilometres travelled over time (e.g. during and outside peak hours). % Fleet composition Number (or share) of vehicles per category and per key characteristic. Categories are e.g. passenger vehicle, truck. Key characteristics are e.g. age, motive power, mass, dimensions, emission factors. Reported per region and per year, or for a specific road (type) at a specific moment in time. Number of vehicles, % of total fleet Modal split Share of each travel or transport mode, per specified time and space of VKT, PKT, TKT. % Impact pathways Impact pathways illustrate how changes caused by CCAM, and assessed in other impact areas, affect the outcomes of this impact area. Mapping these cause-and-effect relationships helps scope the evaluation by grounding it in established theories or empirical findings. It clarifies how localised effects can escalate to larger scale impacts, shows whether one area influences another, and reveals how these changes ripple through vehicles, individuals, the transport system and beyond. Using these pathways can help identify where additional information is needed or where collaboration is required with other impact areas, or to ascertain whether certain aspects fall outside the scope of the assessment. Transport demand and fleet composition are influenced by impacts on people mobility (Chapter 4.2.2), services and operation (Chapter 4.3.1), as well as logistics (Chapter 4.3.2). This is depicted in the impact pathway below (Figure 21). The figure offers a general overview that can be adapted or extended to specific CCAM systems studied. Each project should tailor it by specifying relevant indicators and units. 144 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 24. Traffic safety pyramid (adapted from Hydén [62]). A related concept is subjective safety, which refers to a person’s feeling of safety. The relationship between subjective and objective safety is complex [63]. Typically, better objective safety improves subjective safety, while decreased objective safety worsens it, but mismatches occur. For example, objectively safer conditions may still feel unsafe, while unsafe conditions might feel safe due to a lack of awareness. These changes can lead people to alter their behaviour, for example through risk compensation. In EU-CEM, subjective safety is addressed under user evaluation in Chapter 4.2.1. 4.3.4.2. Guidelines Indicator recommendations Result indicators recommended to be evaluated by each project addressing traffic safety are listed in Table 32 below. Impacts are differences or changes between the baseline and treatment scenarios for these indicators. An impact can be reported as an absolute value (with unit below) or relative value (%). If it is not possible to evaluate some of the indicators below, the reason for this should be reported as part of the ‘comply or explain’ principle of the EU-CEM. The indicators listed below are not exhaustive; in addition to these, each project should define indicators of their own in accordance with Guideline 3.12. Table 32. Indicators recommended to be evaluated by each project addressing traffic safety. Indicator short name Definition Unit Road injuries or injury accidents Number of road injuries or injury accidents, subdivided by severity (fatal, serious, slight). Often reported annually for the target societal scenario and fleet. Number Property damage only accidents Number of property damage only accidents. Often reported annually for the region of the target societal scenario and fleet. Number 145 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Accident risk Frequency of accidents relative to an exposure measure (e.g. vehicle kilometres travelled, hours driven, number of scenario instances). This quantifies how often accidents occur under specified conditions (vehicle type, environment, conflict scenario). Number per exposure unit Accident severity Probability of a given accident severity level (fatal, serious, slight) once an accident has occurred. May be further broken down by environment (e.g. urban/rural/motorway) or scenario type (e.g. merging conflict, lanechange conflict). % Impact pathways Impact pathways illustrate how changes caused by CCAM, and assessed in other impact areas, affect the outcomes of this impact area. Mapping these cause-and-effect relationships helps scope the evaluation by grounding it in established theories or empirical findings. It clarifies how localised effects can escalate to larger scale impacts, shows whether one area influences another, and reveals how these changes ripple through vehicles, individuals, the transport system and beyond. Using these pathways can help identify where additional information is needed or where collaboration is required with other impact areas, or to ascertain whether certain aspects fall outside the scope of the assessment. Traffic safety is influenced by the impacts of CCAM on driving behaviour (Chapter 4.1.2), users (Chapter 4.2.1), people mobility (Chapter 4.2.2), and logistics (Chapter 4.3.2). This is depicted in the impact pathways below (Figure 25). The figure offers a general overview that can be adapted or extended to the specific CCAM systems studied. Each project should tailor it by specifying relevant indicators and units. Figure 25. Input needed from the other impact areas for traffic safety impact assessment and its outputs. 146 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Table 33 provides examples of outputs of traffic safety impact assessment that serve as inputs for other impact areas. Some of these outputs overlap with the recommended indicators listed above, while others must be additionally considered. Please note that the required input needs must be specified in detail, as they are unique to each project. The table below is not exhaustive. Table 33. Outputs from traffic safety impact assessment required as input for other impact areas. Impact area requiring input Needed input Quality of life Equity Number of fatalities and injuries Socio-economics Number of fatalities, injuries, and property damage only accidents Overview of approaches and methods Selecting the right method for evaluating traffic safety ensures relevant and reliable results. The choice is influenced by the project’s objectives and the resources available to the partners. Additionally, the available tools and data shape the feasibility and effectiveness of different approaches (see Chapter 3.3 for further details). Table 34 provides an overview of various approaches used in traffic safety evaluations. It highlights the strengths, weaknesses, and requirements of each method, helping to identify those that best align with the project’s objectives, resources, and specific evaluation aspects. It is important to note that, even when the cons outnumber the pros, one pro can outweigh multiple cons. When analysing surrogate safety measures, the validity of the results also relates to the validity of the measure itself (i.e. it should be proven that the number of conflicts as specified in the study correlates with the number of accidents in the scenario of interest). Simulating safety-relevant scenarios is a common method for assessment, with their main approaches detailed in Table 35. For more information about simulation-based approaches, see ISO 21934 ‘Road vehicles— Prospective safety performance assessment of pre-crash technology by virtual simulation’. Table 34. An overview of approaches and methods for evaluation of traffic safety. Approach/method Pros Cons Requirements Simulation of safety-relevant scenarios Allows systematic exploration of ahigh number of safetyrelevant scenarios. The validity of the result depends on model’s validity, the assumptions made and the input data quality. Software suitable for simulation of safety-critical situations. Road user behaviour model suitable for safety-critical situations. Valid model of the automated vehicle. Suitable injury risk functions to model change in accident severity. 147 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Driving simulator, e.g. for studying human reactions in take-over situations (direct effect) or usage of automated driving systems (indirect effect). Potential to safely explore safety-critical situations. Good control of conditions and repeatability. The validity of the result depends on the simulator’s validity enabling realistic perception of speed and the visual environment. People take more risks in simulators than in real life. Simulator (hardware and software) suitable for testing of safety-critical situations, with potential additional features based on specific scenarios. Analysis based on accident statistics for estimating impacts on fatalities or injuries, or target accidents potentially influenced by CCAM. Allows scaling up to larger regions and longer time periods. Allows estimation of the maximum potential safety impact. Poor data quality and quantity. Lack of contextual accident data to account for e.g. the ODD, various causes or underlying reasons for the accidents. The impact of new systems is not reflected in statistics until they are widely used. Availability of suitable accident statistics. Effect sizes must be estimated in combination with accident statistics. Target accidents must be derived from accident statistics. Table 35. An overview of approaches to simulation of safety-relevant scenarios. Approach Description Pros Cons Baseline approach with Monte-Carlo sampling Simulating all road users with sampling from existing distributions of preconflict situations. Suitable for simulation of safety critical situations. Can consider surrounding traffic. High simulation effort. Distributions of pre-conflict situations must be available. Counterfactual Baseline approach Simulation of variation of realworld accident with different objects: ego-vehicle and conflicting road user(s). Suitable for simulation of safetycritical situations. Not possible to consider surrounding traffic. Limited availability of detailed data of vehicle movements in realworld accident cases. Microscopic traffic simulations and trajectory analysis tool (e.g. SSAM tool [64]) Analysis of conflicts based on trajectories from microscopic traffic simulations. Good software availability. Possibility for scenario testing. Software generally not suitable for simulation of safety-relevant situations, as models typically assume safe driver behaviour and may not accurately capture the complexities of unsafe driving. Unrealistic result if the software is not designed to simulate realistic behaviour in conflict situations. 148 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Pitfalls and best practices Evaluating traffic safety impacts requires consideration of the methods and approaches used, as these can significantly influence the validity and reliability of the results. Overlooking potential pitfalls may lead to flawed conclusions, misinterpretation of findings or unintended biases. By proactively addressing the pitfalls, these can be avoided. Table 36 presents some common pitfalls of evaluation traffic safety evaluation and presents actionable best practices to address them. Table 36. Pitfalls in the evaluation of traffic safety and best practices to avoid them. Topic Pitfall Related best practice Using appropriate simulation tools and models Using tools and models not designed for simulating conflicts or safety-relevant situations, or outside their defined range, can lead to incorrect or invalid outcomes. Use the simulation tool and model developed specifically for studying conflicts or safety-relevant situations, and only in the context they are defined for. Using speed as a surrogate measure for safety Using speed as a safety proxy only from a single-vehicle and not traffic-flow perspective. Lower driving speed is safer only if all vehicles drive slower. Understand how different surrogate measures can and cannot be used. Select a surrogate measure that is valid for the situation. Using conflicts as surrogate measure for safety No validated specifications for conflicts for all traffic environments. Understand how different surrogate measures can and cannot be used. Select a surrogate measure that is valid for the situation. New safety-critical situations Insufficient evidence of the frequency and nature of new safety-critical situations caused specifically by driving automation. Estimate the impact as a function of frequency (per situation type) to show a range of results. If the frequency of the new accidents becomes known later, read the impact from these results. Accident statistics No access to suitable accident databases. Include partners with access to necessary databases in the consortium, or acquire access. 149 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4.3.5. Traffic flow efficiency 4.3.5.1. Introduction Definition and scope EU-CEM defines the traffic flow efficiency impact area as follows: The traffic flow efficiency impact area addresses the impacts of CCAM on collective traffic patterns, travel times and throughput on a road network. Traffic flow efficiency describes the ability of the road network to serve the required demand for travel and transport without unnecessary delays. This chapter covers the effects of CCAM on average travel times and delays of different road user groups, on the throughput of roads, and on the resilience of road networks in terms of recovery after interruptions. Traffic flow characteristics emerge from the behaviours of and interactions between individual drivers and vehicles (driver-vehicle units). The interactions create patterns that are independent of an individual’s behavioural details. This impact area thus concerns the movement of a group of vehicles or the traffic stream as a whole, rather than individual vehicles alone. Not within the scope of this impact area, but linked to it, are impacts on services and operations (see Chapter 4.3.1), people mobility (see Chapter 4.2.2), and logistics (see Chapter 4.3.2). The impacts of CCAM on mobility and logistics service provision influence travel and transport patterns, which in turn affect traffic flow efficiency. Additionally, changes in fleet operations and traffic management (see Chapter 4.3.1), such as platooning or signal control priorities, implemented alongside CCAM systems, may also impact traffic flow efficiency. As a result, the methods used to assess traffic flow efficiency impacts can help identify optimal traffic management or fleet operation measures and parameters. Network theory, which describes the structure and dynamics of networks at different scales, can be combined with traffic flow analysis. For example, changes in network topology (can be addressed under impacts areas of land use (see Chapter 4.4.1) and services and operation (see Chapter 4.3.1)) can enable new links between locations and lead to reallocation of traffic, affecting also travel times on different links. Methods of traffic flow efficiency assessment can be used to plan these changes in topology if traffic flow optimisation is desired. Additionally, impacts on traffic safety (Chapter 4.3.4) directly affect the amount of congestion caused by accidents, but are beyond the scope of this chapter. This chapter does not cover the effects of CCAM on the driving behaviour of individual drivers or vehicles, such as speed preferences, acceleration and deceleration abilities, and headway settings. These are part of the driving behaviour impact area (see Chapter 4.1.2). Impacts on emissions and energy use stemming from changes in traffic flow dynamics are covered in Chapter 4.3.5. Background Traffic flow theory studies the collective movement of vehicles on roads. Collective movement results from interactions between individual driver-vehicle units and their interaction with the road environment and can lead to emergent effects and traffic patterns [65]. The main levels of analysis are microscopic and macroscopic traffic theory, which differ in their scope and approach to representing reality [65]. The microscopic level describes each individual vehicle’s behaviour separately, in terms of (targeted and actual) speed and headway (in space or time), lane changes and braking patterns. On the macroscopic level, traffic flow is analysed as a continuous flow, similar to how liquids behave in motion. Instead of individual vehicles, corresponding variables describe aggregated flow: density in terms of vehicles per road kilometre corresponds to space headway, flow in terms of vehicles per hour to time headway, and average speed to acceleration and deceleration dynamics of vehicles. 150 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Traffic models based on the macroscopic and microscopic, as well as their intermediate mesoscopic, levels have been developed and are commonly applied in traffic simulation software to assess traffic flow efficiency on roads and networks, as well as to ‘predict’ potential changes with different interventions. The microscopic, macroscopic and mesoscopic levels of analysis include changes of the traffic state over time [66]. Another way to describe and analyse traffic flow is by flow-density diagrams, which can be used to study the average behaviour of driver-vehicle units, or to determine the level of service or roads and their capacity (maximum throughput) [65]. A special form of this diagram is the fundamental diagram of traffic flow, which describes the theoretical pairwise relationships between density, flow, and speed in stationary, homogeneous traffic, assuming identical driver-vehicle units [65]. Fundamental diagrams represent the traffic state at a certain road section. A related concept is the macroscopic fundamental diagram, or network fundamental diagram, which represents the relationship between area-wide traffic flow, density, and speed. It can be used when assessing the overall capacity and efficiency of a network. Additionally, resilience theory can be applied to assess a network’s ability to withstand and recover from disruptions caused by external factors such as accidents or weather events. In general, the characteristics of traffic flow on a road or network depend on several elements: • The size and characteristics of vehicles and their variation among vehicles on the road network • Driving behaviour and its variation among vehicles on the road • The demand for road travel and transport distributed over space and time • The road infrastructure and environment • Environmental and other variable conditions. Figure 26 shows the main components of the traffic flow efficiency impact area. Figure 26. Main components of the traffic flow efficiency impact area. 151 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility 4.3.5.2. Guidelines Indicator recommendations The result indicators recommended to be evaluated by each project addressing traffic flow efficiency are listed in Table 37. Impacts are differences/changes between the baseline and treatment scenarios in these indicators. An impact can be reported as an absolute value (with unit below) or relative value (%). If it is not possible to evaluate some of the indicators below, the reason for this should be reported as part of the ‘comply or explain’ principle of the EU-CEM. The indicators listed below are not exhaustive; in addition to these, each project should define indicators of their own in accordance with Guideline 3.12. Table 37. Indicators recommended to be evaluated by each project addressing traffic flow efficiency. Indicator short name Definition Unit Travel time Average travel time per vehicle kilometre travelled (VKT), per vehicle category. Seconds/VKT Delay Difference between actual travel time and travel time according to speed limit. Seconds/VKT Throughput Volume of vehicles per hour or passengers per hour that travel through a given section of a network. Vehicles/h (per lane) or pax/h Travel time reliability Travel time reliability quantifies the variation of travel time for a given trip and time period over a selected time horizon [67]. 50th and 95th percentile of travel time Resilience Resilience is measured as time to recovery, i.e. the time it takes for a traffic system to return to normal or acceptable operating conditions following a disruption, e.g. an accident. The time to recovery starts when the accident site is cleared and vehicles can move again (i.e. the time needed for emergency handling is not counted). Minutes, hours Impact pathways Impact pathways illustrate how changes caused by CCAM, and assessed in other impact areas, affect the outcomes of this impact area. Mapping these cause-and-effect relationships helps scope the evaluation by grounding it in established theories or empirical findings. It clarifies how localised effects can escalate to larger scale impacts, shows whether one area influences another, and reveals how these changes ripple through vehicles, individuals, the transport system, and beyond. Using these pathways can help identify where additional information is needed or where collaboration is required with other impact areas, or to ascertain whether certain aspects fall outside the scope of the assessment. Traffic flow efficiency is influenced by driving behaviour (Chapter 4.1.2), transport activity and fleet composition (Chapter 4.3.3), and services and operation (Chapter 4.3.1). This is depicted in the impact pathway in Figure 27. The figure offers a general overview that can be adapted or extended to specific CCAM systems studied. Each project should tailor it by specifying relevant indicators and units. 152 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Figure 27. Input needed from the other impact areas for traffic flow efficiency impact assessment and its outputs. Table 38 provides examples of outputs of the impact assessment of traffic flow efficiency that serve as inputs for other impact areas. Some of these outputs overlap with the recommended indicators above, while others must be additionally considered. Please note that the required input needs must be specified in detail, as they are unique to each project. The table below is not exhaustive. Table 38. Outputs from traffic flow efficiency impact assessment required as input for other impact areas. Impact area requiring input Needed input Services and operation Logistics Travel time and its predictability Traffic flow efficiency Travel time, Delay Energy and environment Speed patterns Accessibility Travel time per mode Overview of approaches and methods Selecting the right method for evaluating traffic flow efficiency ensures relevant and reliable results. The choice is influenced by the project’s objectives and the resources available to the partners. Additionally, the available tools and data shape the feasibility and effectiveness of different approaches (see Chapter 3.3 for further details). 153 European Common Evaluation Methodology Handbook for Connected, Cooperative and Automated Mobility Traffic simulation is a widely used tool for analysing traffic flow efficiency and assessing changes resulting from different interventions, as it enables cost-effective study of the movements of large numbers of vehicles over a road or network. Impacts of changes to any of the elements, such as driving behaviour, road layout or traffic management measures on traffic flow characteristics, can be estimated. In line with the microscopic and macroscopic traffic flow theories, different simulation models have been developed with different application scopes. Microscopic models describe trajectories of individual vehicles over space and time. They can, to some extent, model vehicles with a range of driving behaviours, and may therefore be suitable for studying the impacts of driving automation or connectivity on traffic flow efficiency [65]. Microscopic traffic simulation typically considers road sections or small networks, simulating single traffic scenarios. If the societal scenario requires assessing impacts on a regional scale, results from these traffic scenario simulations can be scaled up to the target region using available statistics and other data. Macroscopic traffic models consider collective patterns across a network. These models can be used to estimate effects on the societal scenario directly, provided that a suitable model and required input are available on the macroscopic level of potential changes resulting from different microscopic behaviour. The mesoscopic level combines elements of the microscopic and macroscopic viewpoints. Several questions related to CCAM services may require trip-based dynamic traffic models, which allow for changing mode and route while considering a traveller’s full day of movement. They also account for different modes and CCAM services for various trips throughout the day. It is important to note that, since traffic is always influenced by human behaviour and external conditions, no model can achieve perfect accuracy. It is, therefore, necessary to be clear about the assumptions and limitations when interpreting the results of simulation studies, while making sure to employ the theories and models with best descriptive power for the research questions and scenarios set in the project [68]. This is especially true for CCAM systems that are not yet widely deployed, and several assumptions are required when implementing their behaviour into traffic models. Table 39 provides an overview of various approaches used in traffic flow efficiency evaluation. It highlights the strengths, weaknesses, and requirements of each method, helping to identify those that best align with the project’s objectives, resources, and specific evaluation aspects. It is important to note that, even when the cons outnumber the pros, one pro can outweigh multiple cons. Table 39. An overview of approaches and methods for evaluation of traffic flow efficiency. Approach/method Pros Cons Requirements Field experiment Enables direct observation of the effects of CCAM on traffic flow, if the penetration rate under evaluation can be tested. Feasible in use cases where the number of CCAM vehicles is low also in the targeted societal scenario, such as public transport services. Can be used as a way to collect input for calibration Quantitative observations of some traffic phenomena are challenging to reproduce consistently [67]. Requires significant effort. Often, a sufficient penetration rate of automated driving in Real-world implementation of CCAM system of sufficient TRL. Clear view on how tested situations link to traffic flow efficiency impacts. Similar baseline and treatment conditions (with respect to traffic