D1.6 DevOps methodology and CI/CD strategy for EMERALD- v2

Deliverable D1.6 DevOps methodology and CI/CD strategy for EMERALDv2 Editor(s): Gorka Benguria, Iñaki Etxaniz (TECNALIA) Responsible Partner: TECNALIA Research and Innovation Status-Version: Final - v1.0 Date: 30.04.2025 Type: R Distribution level (SEN, PU): PU D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 2 of 54 www.emerald-he.eu Project Number: 101120688 Project Title: EMERALD Title of Deliverable: D1.6 DevOps methodology and CI/CD strategy for EMERALD-v2 Due Date of Delivery to the EC 30.04.2025 Work package responsible for the Deliverable: WP1 - Concept and methodology of EMERALD Editor(s): Gorka Benguria Elguezabal, Iñaki Etxaniz (TECNALIA) Contributor(s): Gorka Benguria Elguezabal, Iñaki Etxaniz (TECNALIA) Reviewer(s): Franz Deimling (FABA) Cristina Martínez, Juncal Alonso (TECNALIA) Approved by: All Partners Recommended/mandatory readers: WP1, WP2, WP3, WP4, WP5 Abstract: Final version of the description of the DevOps methodology and CI/CD strategy that provides details on the integration process followed to create and deploy the integrated EMERALD CaaS (Compliance as a Service) Framework. It also provides details on the strategies applied at integration and deployment level to help on the achievement of the EMERALD goal. Keyword List: DevOps, CI/CD, Integration, Container, Environment, Releases Licensing information: This work is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0 DEED https://creativecommons.org/licenses/by-sa/4.0/) Disclaimer Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. The European Union cannot be held responsible for them. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 3 of 54 www.emerald-he.eu Document Description Version Date Modifications Introduced Modification Reason Modified by v0.1 15.04.2025 First draft version TECNALIA v0.2 17.04.2025 Typos corrected and style polished. Sent for internal QA review TECNALIA v0.3 17.04.2025 QA Review Franz Deimling (FABA) v0.4 26.04.2025 Addressed all comments received in the Internal QA review TECNALIA v0.5 28.04.2025 Final review TECNALIA v1.0 30.04.2025 Submitted to the European Commission TECNALIA D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 4 of 54 www.emerald-he.eu Table of contents Terms and abbreviations ............................................................................................................... 7 Executive Summary ....................................................................................................................... 8 1 Introduction ........................................................................................................................... 10 1.1 About this deliverable .................................................................................................. 10 1.2 Document Structure .................................................................................................... 11 1.3 Updates from D1.5....................................................................................................... 11 2 DevOps Methodology ............................................................................................................ 14 2.1 Context ......................................................................................................................... 14 2.2 Goals ............................................................................................................................ 14 2.3 Processes ..................................................................................................................... 15 2.3.1 Plan .................................................................................................................... 16 2.3.2 Code ................................................................................................................... 17 2.3.3 Build ................................................................................................................... 18 2.3.4 Test .................................................................................................................... 18 2.3.5 Release............................................................................................................... 19 2.3.6 Deploy ................................................................................................................ 20 2.3.7 Operate .............................................................................................................. 20 2.3.8 Monitor .............................................................................................................. 21 2.4 Lifecycle ....................................................................................................................... 21 3 CI/CD Strategy ........................................................................................................................ 24 3.1 CI Strategy .................................................................................................................... 24 3.1.1 Container-based ................................................................................................ 24 3.1.2 Environments with IaC ...................................................................................... 25 3.1.3 Integration guidelines ........................................................................................ 26 3.1.4 CI/CD Components ............................................................................................ 26 3.1.5 Component-based Kustomize ........................................................................... 29 3.1.6 Manual deployment support ............................................................................. 30 3.1.7 Rancher for debugging support ......................................................................... 32 3.1.8 Local environment for testing ........................................................................... 32 3.1.9 Progressive Verification ..................................................................................... 32 3.1.10 Automation ........................................................................................................ 33 3.2 CD Strategy .................................................................................................................. 33 3.2.1 Releases ............................................................................................................. 34 3.2.2 Public Assets Release ......................................................................................... 34 3.2.3 Keycloak configuration ...................................................................................... 35 3.2.4 Demo pilot ......................................................................................................... 35 D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 5 of 54 www.emerald-he.eu 3.2.5 Documentation .................................................................................................. 35 3.2.6 Environments with IaC ...................................................................................... 36 3.2.7 Automation ........................................................................................................ 36 4 Conclusions ............................................................................................................................ 37 5 References.............................................................................................................................. 38 APPENDIX A: Project Risks and impact in the DevOps Methodology ......................................... 40 APPENDIX B: Project Milestones from the DoA .......................................................................... 42 APPENDIX C: Integration files ...................................................................................................... 43 C.1 – Renovate mechanism ................................................................................................. 43 C.2 – Semantic Versioning Configuration ............................................................................ 44 C.3 – Kustomize approach ................................................................................................... 45 C.4 – Docker compose approach ......................................................................................... 48 C.5 – CI/CD Examples ........................................................................................................... 50 List of figures FIGURE 1. DEVOPS CYCLE ................................................................................................................. 16 FIGURE 2. LIST OF ISSUES RELATED TO CONCEPT & METHODOLOGY IN EMERALD ..................................... 22 FIGURE 3. A MERGE REQUEST RELATED TO CONCEPT & METHODOLOGY IN EMERALD .............................. 22 FIGURE 4. A MERGE REQUEST MECHANISM TO PRODUCE A NEW RELEASE IN EMERALD............................. 34 FIGURE 5. RENOVATE SCHEDULE ........................................................................................................ 44 FIGURE 6. KUSTOMIZE MAIN STRUCTURE OF THE INTEGRATED CAAS FRAMEWORK ...................................... 45 FIGURE 7. KUSTOMIZE COMPONENT ................................................................................................... 47 FIGURE 8. DOCKER COMPOSE FRAMEWORK ......................................................................................... 49 FIGURE 9. LOCAL ENVIRONMENT SERVICES ........................................................................................... 50 FIGURE 10. RCM CHANGE ................................................................................................................ 50 FIGURE 11. DOCKER CI/CD EXAMPLE ................................................................................................. 51 FIGURE 12. DOCKER CI/CD EXAMPLE PIPELINES .................................................................................... 52 FIGURE 13. DOCKER CI/CD STAGES DETAIL .......................................................................................... 52 FIGURE 14. SEMANTIC RELEASE CI/CD EXAMPLE .................................................................................. 53 FIGURE 15. SEMANTIC RELEASE CI/CD STAGES DETAIL ........................................................................... 54 List of tables TABLE 1. OVERVIEW OF DELIVERABLE UPDATES WITH RESPECT TO D1.5 .................................................... 12 TABLE 2. RISK AND MITIGATION LIST ................................................................................................... 40 List of listings LISTING 1. CI/CD FOR THE RCM COMPONENT ..................................................................................... 29 LISTING 2. CI/CD FOR SIDE-SERVICE ................................................................................................... 29 LISTING 3. EXAMPLE COMMANDS TO MANUALLY REDEPLOY A COMPONENT .............................................. 31 LISTING 4. RENOVATE PIPELINE (.GITLAB-CI.YML) .................................................................................. 43 LISTING 5. CONTENT OF THE FILE RENOVATE.JSON ................................................................................. 44 D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 6 of 54 www.emerald-he.eu LISTING 6. CONTENT OF THE FILE .RELEASERC.YAML ............................................................................... 45 LISTING 7. KUSTOMIZE INTEGRATE OVERLAY ......................................................................................... 46 LISTING 8. KUSTOMIZE BASE .............................................................................................................. 46 LISTING 9. KUSTOMIZE RCM COMPONENT .......................................................................................... 48 LISTING 10. CI/CD FOR DOCKER GENERATION ...................................................................................... 51 LISTING 11. CI/CD FOR SEMANTIC RELEASE GENERATION ....................................................................... 54 D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 7 of 54 www.emerald-he.eu Terms and abbreviations AI Artificial Intelligence AI-SEC AI Security Evidence Collector AMOE Assessment and Management of Organizational Evidence API Application Programming Interface CaaS Compliance-as-a-Service1 CI/CD Continuous Integration / Continuous Deployment CLI Command Line Interface CMMI Capability Maturity Model Integration DevOps Development and Operation DIND Docker in Docker DoA Description of the Action EC European Commission EUCS European Cybersecurity Certification Scheme for Cloud Services GA Grant Agreement to the project HTTP Hypertext Transfer Protocol IaC Infrastructure as Code IEC International Electrotechnical Commission ISO International Organization for Standardization ITIL Information Technology Infrastructure Library K8s Kubernetes K8so Kubernetes on Openstack K8sv Kubernetes on Vsphere MARI Mapping Assistant for Regulations with Intelligence OSCAL Open Security Controls Assessment Language RCM Repository of Controls and Metrics TWS Trustworthiness System UI/UX User Interface / User Experience WP Work Package 1 Please note that in previous deliverables and in the DoA, the term Certification-as-a-Service was used to stand for CaaS. Compliance has now been introduced to clarify that EMERALD can be used to assess both normative models and internal organizational models. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 8 of 54 www.emerald-he.eu Executive Summary The main objective of EMERALD project is to provide a framework that enables continuous compliance and certification, and agile lean re-certification to consume services that adhere to a defined level of security and trust in a uniform way across heterogeneous environments made of combinations of various resources. In order to achieve the development of the CaaS (Compliance-as-a-Service) 2 Framework, the EMERALD project aggregates specialized components from different teams and integrates them into a single framework that is validated in different pilots. This brings some challenges that should be managed: on the provider side, the components are developed by different teams with different schedules and different development practices; on the consumer side, the CaaS Framework will be consumed as a service or as an on-premises configuration. This document is the final version of the DevOps (Development and Operations) Methodology and the CI/CD (Continuous Integration/Continuous Deployment) strategies. The previous version, presented at M6, focused mainly on stablishing the corresponding processes and tools to achieve the first integrated version of the CaaS Framework (M18). This final version is the result of the experience and lessons learnt during the first half of the project and focuses on the upcoming releases of the CaaS Framework (M30 and M34) and the deployment of the CaaS Framework in the pilots. The main changes that have been introduced relate to the following aspects: • Integration of new versions of the CaaS Framework: Supporting, on the one hand, the agile integration of components developed by the different teams and, on the other hand, the validation of the upcoming versions of the CaaS Framework before being deployed in the pilots with a certain degree of confidence. • Delivery of new versions to the Pilots: Supporting the deployment of the validated versions of the CaaS Framework in the pilots and introducing mechanisms to reduce the undesirable effects of deployment in their environments (such as data loss, the need to reconfigure the environment or service downtime, to name a few). • Getting and processing feedback: Supporting the collection of feedback from the pilots and the development teams, and the processing of that feedback to improve the CaaS Framework. The target audience of the document are the EMERALD participants in charge of coordinating the development and operation activities. In addition, the document also aims to provide information to other partners and stakeholders in understanding how the EMERALD CaaS Framework is managed from a DevOps perspective. The document also includes annexes that provide additional information about the project, which can help the reader grasp the project context. In particular, they include the risks and milestones defined in the Description of Action (DoA) and details on the integration files that are not accessible in the public area of the GitLab repository of the EMERALD project. This document is the second and final version of the development and operation coordination approach that has been applied during the first half of the project. Future related work will involve applying the techniques defined here to develop the v2 (M30) and v3 (M34) releases of the CaaS Framework, as well as to the deployments in the pilots. Some changes to the 2 Please note that in previous deliverables and in the DoA, the term Certification-as-a-Service was used to stand for CaaS. Compliance has now been introduced to clarify that EMERALD can be used to assess both normative models and internal organizational models. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 9 of 54 www.emerald-he.eu methodology are likely to be introduced, but the fundamental processes are already defined and in operation. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 16 of 54 www.emerald-he.eu • Code: During this process, the integration of the different components is encoded as a container orchestration code (aka choreography). In addition, code for the environment and side-services can be developed, when needed. • Build: The building process is automated. It takes the choreography and deploys the CaaS Framework in the development environment. • Test: The test process focuses on the integration testing. It covers the development of the necessary tests to verify the behaviour of the framework in the long term. • Release: We create a formal release of the CaaS Framework for each release milestone planned, as well as when updates in components require it. • Deploy: The deploy process is automated. It takes the choreography and deploys the tagged CaaS Framework in the production environment, and the pilot environments are informed about the new version. • Operate: Apart from the regular operation, this process is focussed on the validation management. • Monitor: The monitoring of the production environment is adapted in this stage to cover the production and pilots’ environments. Figure 1. DevOps Cycle We take these processes as a basis for our DevOps Methodology, adapting them to the specificities of the EMERALD project. For each process, we provide below a brief description and the expected inputs and outputs. 2.3.1 Plan During the planning process, we collect the integration and deployment needs of the project, prioritize the activities to be performed, and track their completion. This activity is performed on demand and periodically. Whenever a need over the integration is identified or received, it is collected, and a quick prioritisation is performed. In case the need is urgent, it is put into implementation. Otherwise, it is put into the backlog of needs to be performed in the next iterations. Besides, every two weeks, in a meeting with the development teams and the pilots, we collect their needs and prioritise them. That meeting is also used to review the status of the tasks that are being worked out in the DevOps team. The starting point for the needs of the project was the project DoA [3], where the EMERALD overall needs were described, and then the requirements gathered in the project architecture deliverables, D1.3 [5] and D1.4 [6]. In addition, we have collected the needs from different sources: • Mainly, the periodic meetings with the development teams • Messaging channels (e.g., Teams) D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 17 of 54 www.emerald-he.eu • The EMERALD WP1 mailing list (WP1 is in charge of the integration) • The issue mechanism in GitLab The inputs and outputs of this process can be summarised as follows: • Inputs: o Requirements o Feedback from pilots’ validation o EMERALD Architecture o External service requirements • Outputs: o Summary of the activities performed in the last cycle o Activities to be performed in the next cycle o Activities backlog o Issues linked to the requirements. 2.3.2 Code During the coding process, the integration of the different components is coded in a container orchestration code (aka choreography). In addition, code for the environment and side-services can be developed when needed. Once the development team has packaged or updated a component, a collaboration cycle with the DevOps team starts. During this cycle, the DevOps team supports the development team in different areas: • Repository creation: The DevOps team creates a repository for each component in the EMERALD GitLab repository and provides the necessary access to the development team to manage their own component group. • Image creation: The DevOps team provides support to the development team in the creation and publishing of the Docker image (or images) for the component. On the one hand, providing the EMERALD Artifactory repository (implemented in Artifactory 3 ) to store the Docker image and, on the other hand, providing support in the creation of the Dockerfile using CI/CD GitLab components. • Orchestration creation: A Kustomize 4 -based orchestration has been created to help the development teams to create the orchestration code. The baseline orchestration has been created in a modular way to allow the development teams to add and test their components easily. • Debugging support: Different resources are provided by the DevOps team to help the development teams to debug their components. We started with the logging and the console access to the Kubernetes cluster. Later, based on the feedback from the development teams, we added more debugging resources to the development teams such as the Keycloak testing environment, or the local testing environment based on Docker Compose. • Update Support: Guidelines and automatic update mechanisms are provided to the development teams to help them update their components in the CaaS Framework. The guidelines are provided in the EMERALD GitLab repository, and the automatic update mechanisms are implemented with Renovate 5 and GitLab CI/CD. • Adding side-services: The DevOps team helps with the creation of side-services that are needed to support the integration of the components. For example, adding administration frontends for databases to help the development teams in the analysis 3 https://jfrog.com/artifactory/ 4 https://kustomize.io 5 https://github.com/renovatebot/renovate D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 18 of 54 www.emerald-he.eu of the data contained. On the other side, the development teams are responsible of applying the guidelines provided by the DevOps team to integrate and update their components in the CaaS Framework. The main outcome of this process is a Kustomize component for each of the components. The inputs and outputs of this process can be summarised as: • Inputs: o Components from the different development teams in a package repository (in EMERALD, implemented in Artifactory) • Outputs: o Choreography and configuration code for the integration environment. o Infrastructure as code (IaC) for the environments. 2.3.3 Build The building process creates a merged manifest for a specific Kustomize overlay. On the one hand, it can be done manually – by the DevOps or the development team – to generate a combined manifest to deploy the CaaS Framework. On the other hand, it can be done automatically as part of the CI/CD pipelines. The inputs and outputs of this process can be summarised as: • Inputs: o Packages from the components o Choreography code and configuration for the integration environment o Infrastructure as code (IaC) for the environments • Outputs: o Integration environment o CaaS Framework in the integration environment. 2.3.4 Test The test process covers several activities, such as the update of the candidate production environment based on the development environment, the creation of the integration tests, and the application of the integration tests over the candidate production environment. First, the deployment of the candidate production environment is done automatically, together with the build process, in a CI/CD pipeline. Then, for the creation of the integration tests, three steps are performed: • First, the DevOps team creates the integration tests that are needed to verify the functionality of the evidence collector components. These components are not deployed as part of the CaaS Framework services. Instead, they are deployed on the resources to be certified, in order to collect evidence of their compliance. Therefore, we need this mechanism to be able to verify the compatibility of the collectors with the CaaS Framework. • Second, the DevOps team creates the integration tests covering the EMERALD workflows, as defined in D4.2 [20]. • Third, the DevOps team implements persistence tests to verify that the data in the CaaS Framework is not lost during the deployment of new versions of the CaaS Framework. Finally, the integration tests are to be applied over the candidate production environment. The DevOps team provides the necessary resources to deploy the candidate production environment and to run the integration tests. The objective of the tests is to provide some D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 19 of 54 www.emerald-he.eu confidence on the correctness of the integration of the components before deploying them into the intended pilots. The inputs and outputs of this process can be summarised as: • Inputs: o CaaS Framework in the integration environment o List of collector components and their usage procedures o Main workflows defined as part of the D4.2 [20] • Outputs: o Integration tests o Integration test results. 2.3.5 Release During the planned release milestones, as well as when other releases require it, we will create a formal release of the CaaS Framework. There are two kinds of releases: • Milestones releases: These are planned and will take place at M18, M30 and M34. • Minor and fix releases: These may take place at any point based on the evolution of the components. On the planned release milestones, the DevOps team creates a release commit over the CaaS Framework repository. This commit triggers a Release CI/CD pipeline that creates the candidate production environment and runs the integration tests over it. The results of the integration tests are used to decide if the release is successful or not. The EMERALD project applies semantic versioning [31] to the releases of the CaaS Framework. The planned releases apply a mayor upgrade (e.g., v1.0.0 to v2.0.0) to the version of the CaaS Framework. Other releases are triggered by the development teams when they update their components. In this case, we use an automatic approach to introduce the new version of the component in the CaaS Framework. The automatic approach is based on the Renovate tool. This tool scans periodically (every week) changes in the images used in the manifests that compose the CaaS Framework. When a new version of the image is detected, it creates a merge request with the new version of the component. This merge request is automatically tested against the integration tests. If the integration tests are successful, the merge request is automatically merged into the CaaS Framework repository and a patch upgrade (e.g., v1.0.0 to v1.0.1) is applied to the version of the CaaS Framework. The inputs and outputs of this process can be summarised as: • Inputs: o Release request o Integration tests • Outputs: o Integration test results o CaaS Framework formal release. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 20 of 54 www.emerald-he.eu 2.3.6 Deploy This process deploys the formal release of the CaaS Framework in the production environments. We support two types of deployment: • SaaS (Software as a Service): The CaaS Framework is deployed in the EMERALD infrastructure and the pilots consume it as a service. In this case, the deployment is done automatically as part of the Release CI/CD pipeline. • On-premises: The CaaS Framework is deployed in a pilot infrastructure and the pilot consumes it as an on-premises configuration. In this case, the deployment is done manually. The inputs and outputs of this process can be summarised as: • Inputs: o CaaS Framework formal release • Outputs: o Production environment as SaaS o Pilot environments, if required. 2.3.7 Operate Besides the regular operation activities that are performed in the production environment, as for example: • Capacity management • Performance management • Availability management • Incident management • Backup management We also include in this stage the validation process, which is not performed by the DevOps team, but by the pilots and the development teams. Please note that the DevOps team is responsible for the validation of the CaaS Framework. Two main aspects are covered in the Operate process: • Gathering the feedback from the pilots and the development teams. • Processing the feedback to improve the CaaS Framework. Feedback is gathered through the EMERALD project communication channels. Mainly the periodic meetings with the development teams and the pilots, and the messaging channels (e.g., Teams). Feedback is processed by the DevOps team and the development teams and transformed into pending tasks in the backlog of the DevOps team. These tasks are prioritised and at given points they become into issues in the GitLab repository. The issues are then planned in the next iteration of the DevOps by the development teams. The inputs and outputs of this process can be summarised as: • Inputs: o Tagged CaaS Framework in the production environment (and pilot environments, if required) • Outputs: o Feedback from pilots’ validation. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 21 of 54 www.emerald-he.eu 2.3.8 Monitor The monitoring of the integration environment is adapted in this stage to cover the production and pilots’ environments. Based on the feedback from the pilots’ monitoring, mechanisms are created to check the behaviour of the framework in the long term. The inputs and outputs of this process can be summarised as: • Inputs: o Tagged CaaS Framework in the production and pilot environments, if required • Outputs: o Monitoring procedures o Monitoring results. 2.4 Lifecycle There are several applicable lifecycles in software development [22], [23], but considering the following characteristics of the EMERALD project: • There are different components with different set of requirement, different teams, and different agendas. • There are fixed milestones (see APPENDIX B: Project Milestones from the DoA) at project level that should be achieved. We decided to apply an iterative process, as stated in the DevOps lifecycle (see Figure 1). The DevOps iterations are continuous and are performed whenever the DevOps team receives integration and release requests from the development teams. Besides those requests, the DevOps team also performs iterations every two weeks, focussing on the environment’s setup, integration tests, monitoring mechanisms, and monitoring results. We use three mechanisms to document the tasks to be performed (e.g., implement monitoring service in k8sv) in the context of the DevOps activities: tasks, issues, and merge requests. • Tasks are used in the context of the periodic meetings with the development teams and the pilots to document the tasks that are going to be performed in the next iteration. They are also used to document the tasks that will be performed afterwards, this is what we call the backlog of tasks. • Issues 6 are used as the primary mechanism for documenting tasks that involve some effort on the part of the DevOps team. Every task is documented in an issue inside the affected repos under the DevOps group in the project GitLab repository (see Figure 2). • Merge requests 7 are used change the code in order to support the completion of the issues created (see Figure 3). 6 https://docs.gitlab.com/ee/user/project/issues/ 7 https://docs.gitlab.com/ee/user/project/merge_requests/ D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 22 of 54 www.emerald-he.eu Figure 2. List of Issues related to Concept & Methodology in EMERALD In the case of issues and merge requests, they must be related with some requirement(s) in order to keep track of its implementation in the DevOps activities. Issues are linked with the requirements using the “linked requirements” mechanism provided by GitLab. In the case of merge requests, if they have not been created from an issue, they will be related with an issue in the description and that issue will be related with some requirement(s). Figure 3. A Merge Request related to Concept & Methodology in EMERALD Tasks are managed in an agile approach, with some flexibility. For example, if there is an urgent need to quickly update components at the request of the development teams they will be done D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 23 of 54 www.emerald-he.eu directly. Regular tasks enter the DevOps lifecycle as requests in the task backlog (i.e., as issues in the DevOps repository). At the end of each iteration (i.e., bi-weekly), we perform an internal meeting in WP1 to: • Review the status of completion of the tasks worked out during the iteration. • Choose the new set of tasks to be carried out during the next iteration. The status of the DevOps tasks is visible at the DevOps group level in GitLab and is recorded in the minutes of the periodic WP1 meetings with the development teams and the pilots. The execution of each task may involve one or more planned processes (except for the monitoring process that is automated and continuous), such as Code, Build, Test, Release, Deploy, and Operate. These processes are not expected to be carried out in every task, i.e., depending on the nature of the task, it may involve all or some of them, and the effort in each task varies. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 24 of 54 www.emerald-he.eu 3 CI/CD Strategy This section provides details on the strategies and technical approach applied during the development of the first version of the CaaS Framework. Besides, we also outline some strategies that will be matured during the next stages of the project, such as those that will support the upcoming releases of the CaaS Framework, as well as the deployment of the CaaS Framework in the pilots. The section is divided into two main parts: CI Strategy and CD Strategy. The description contains much information in common with D1.5 [1] with the final aim of providing a self-contained section that facilitates the reader's understanding. 3.1 CI Strategy For the integration of the outcomes of the development teams in the CaaS Framework, we have applied the following technological approaches: • Components packaged as containers • Environment defined with Infrastructure as Code (IaC) • Integration guidelines • CI/CD Components • Component based Kustomize • Manual deployment support • Rancher for debugging support • Local environment for testing • Progressive verification • Integration automation. 3.1.1 Container-based Container technology has proved to be a very good approach to aggregate components from different teams. Besides, if used appropriately, it also provides de facto scalability and resilience when we use container orchestration technologies, such as Kubernetes 8 or Docker Swarm 9 . In addition, the usage of the container technology promotes decoupling from the architecture which provides some benefits over monolithic architectures [24], [25]. EMERALD prioritises container images as the default packaging technology for its components. In case some components cannot be deployed as containers, IaC and service approaches are prioritized as backup strategies. As a container technology, we have used the Docker ecosystem to build and share images. For image building we support both Docker and Docker Compose. A component of the EMERALD architecture may include one or more dockerfiles 10 to build the images that are used to deploy the CaaS Framework. In some cases, the building process may require some orchestration. In those cases, Docker Compose or Custom Scripting is used as well. To support both scenarios as part of the DevOps activities, we have provided resources and support for the automation of building such images. We have used different technologies including: • Docker shared runners that support Docker in Docker (dind) 11 technology 8 https://kubernetes.io/ 9 https://docs.docker.com/engine/swarm/ 10 https://docs.docker.com/build/concepts/dockerfile/ 11 https://www.docker.com/resources/docker-in-docker-containerized-ci-workflows-dockercon-2023/ D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 25 of 54 www.emerald-he.eu • Kubernetes group runners • Docker machine shared runners that support dind technology As an IaC technology, OpenTofu 12 and Ansible 13 have been favoured, since both technologies are open-source and facilitate knowledge sharing and latter distribution of the resources. Finally, in case there are some components that cannot be physically deployed in production and must be consumed as a service, the OpenAPI 14 Specification is promoted. Regarding the strategy with respect to the packaging on behalf of the DevOps team, it is planned to work as follows: • Provide example packaging approaches starting from a Docker or a Docker Compose specification. These examples are specific to the GitLab framework used in the EMERALD project and include: o .gitlab-ci.yml for Docker and gitlab-ci.yml for Docker Compose. o Example of a dockerfile. o Example of a docker compose file. o Readme file with indications on how to integrate the component. • Each example includes guides on how to integrate it on the components that are stored in the GitLab framework used in the EMERALD project. • When necessary, WP1 has provided support in the integration of the gitlab-ci, and in the development of Docker and Docker Compose. This has been managed through a merge request from the interested party, which has been documented in an issue related to a project requirement. 3.1.2 Environments with IaC The environments that support the CaaS Framework integration and validation were developed following an IaC approach with state of the practice open-source tools. The integration environment has been developed following this approach. For that, a project has been created in the private GitLab of EMERALD (emerald/private/devops/opentofuk8sv). This project creates a three-node Kubernetes cluster over vSphere platform and includes instructions to replicate the deployment on another vSphere platform if necessary. In addition, it uses a reusable set of Ansible playbooks to configure the EMERALD Kubernetes that have been applied in this case and could also be applied on other nodes with little or no customization. The IaC is configurable through the modification of two templated yaml files: • Base OpenTofu hosts: (/blob/master/base_opentofu_hosts.yaml.erb), that controls the initial creation of the virtual machines and their configuration by Ansible. • Ansible host: (/blob/master/Ansible_hosts.yaml.erb), that configures the machines using some Ansible-playbooks. The same approach was followed for the instantiation of other environment and resources, so that we can replicate them in case we need to do so in latter stages of the project. With the project already started (M7), the integration environment was moved from vSphere to OpenStack. In this migration we forked the IaC (emerald/private/devops/opentofu-k8sv) into a new project (emerald/private/devops/opentofu-k8so). In this project we added the 12 https://opentofu.org/ 13 https://docs.ansible.com/ 14 https://www.openapis.org/ D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 32 of 54 www.emerald-he.eu • Execute a command into the component (kubectl exec) • Check the RCM API internally (curl). To do so we create a temporal component in Kubernetes to check access to services from inside the cluster (kubectl run --rm) 3.1.7 Rancher for debugging support Besides the manual deployment support, we also provide a Rancher-based web interface to manage the Kubernetes cluster. This is convenient for many reasons: • Provides a user management interface • Allows to quickly check the status of the deployments • Provides a fast way to check the logs of the pods User management is necessary in EMERALD because we try to simulate a realistic deployment scenario, and it is not realistic to have an admin access when deploying the CaaS Framework in the pilots. The Rancher user management has been used to create a highly constrain emeralddeveloper user, which has only write-access on the emerald-dev project and read-access to the Keycloak project for debugging purposes. Besides, it is also very useful to have quick access to monitoring and debugging tools. It is very common in the DevOps team to be inquired about the status of some aspect of the CaaS Framework (e.g., the status of a component, the status of a deployment, or the status of a pod). Initiating the developer CLI requires time and sometimes it may be more complex than expected. In these cases, the Rancher interface provides a quick access to this information. The only things required are the URL of the Rancher interface (https://k8so.emerald.digital.tecnalia.dev/) and the user credentials. 3.1.8 Local environment for testing A partial local environment based on Docker Compose has been created. This environment is not intended to be used in production, but it is useful for some development and debugging situations. It provides a faster mechanism to test interaction between components. This local environment has been used in two situations: • To understand the configuration of the consul service required by the RCM component. • To understand the configuration of Keycloak for several components. The RCM local test requires several side-services to be able to run (i.e., a database and a consul service). These services require some initial configuration that must take place before the RCM modules are started. In this situation, the local environment was used to understand, sequence and test that configuration. Besides, the environment has shown to be useful not only to understand how to configure them but also to deploy them locally. Currently the RCM developers use the local environment when they develop the RCM component. On the other hand, one of the challenges for the DevOps team is to understand the configuration of the Keycloak service required by the EMERALD components. One of the lessons learn from previous projects was that Keycloak configurations must be stored under version control. This enables to configure the Keycloak service before the EMERALD components are started. The local environment was used to understand, sequence and test that configuration. 3.1.9 Progressive Verification The verification of the added and updated components of the CaaS Framework is an important aspect to ensure the secure evolution of the platform during the project. The verification is covered by a set of integration tests that are being automated. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 33 of 54 www.emerald-he.eu Verification activities focus on multiple aspects: • Establish the means to check the health of the components • Verify the compatibility of the evidence collector components • Define more complex integration tests based on EMERALD workflows In the first stage of the project, as components are added, procedures should be implemented to check their health. These mechanisms are used during the integration tests to be performed after the update of each component, as well as during continuous monitoring. As the project advances, additional integration tests are added, based on requests from the developers and feedback from the pilots. The strategy in EMERALD is to document verification-focused activities as issues linked to the requirements. Implementing means to check the health of a component may require implementing parts in the choreography and parts in the component itself. More complex integration may require implementing specific components to generate the activity required to verify such complex integration scenarios. 3.1.10 Automation DevOps activities focus on the automation of all the activities related to the evaluation of components as they are updated by the developers. The main focuses of automation are: • Update the integration platform as the developers update the components. • Run the integration test as the platform is updated. • Update the monitoring mechanism to measure the health of the CaaS Framework in the long term. The strategy is to use GitLab Agent 21 for Kubernetes, implemented in the DevOps repository inside the GitLab repository of EMERALD (emerald/private/devops/gitlab-agent-k8so). It monitors the CaaS Framework repository, and every change detected there is translated into the aimed environments, which allows to deploy new component versions directly, without integration testing. This is done to speed up the feedback to the development team. Integration tests are started at a later stage, using a GitLab runner. The monitoring mechanism is updated following the same approach as the CaaS Framework, i.e., we use the GitLab Agent for Kubernetes for this purpose as well. 3.2 CD Strategy For the deployment of the CaaS Framework to be evaluated by the project and the pilots, the following technological approaches are applied: • Releases • Public Assets Release • Keycloak Configuration • Demo pilot • Documentation • Environment defined with IaC • Deployment automation 21 https://docs.gitlab.com/user/clusters/agent/install/ D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 34 of 54 www.emerald-he.eu 3.2.1 Releases The project follows a versioning system with three mayor releases: • v1.0.0 - First release of the EMERALD components in month 18 • v2.0.0 - Second release of EMERALD components in month 30 • v3.0.0 - Final release of EMERALD integrated audit suite in month 34 Additional releases are expected between those mayor releases, as the project advances and the CaaS Framework is validated. Versions v1.x.x have been created during the first iterations of the project before the month 12. Versions v3.x.x are also expected based on the feedback of the last validation activities. The strategy for the releases starts with a petition from an EMERALD partner (see Figure 4). Basically, the release consists of moving the content from the integration branch to the production branch. To perform this process an issue is created describing the request and the purpose. From that issue a new draft merge request is created over the production branch. That creates a new working branch, where we move the integration version that we want to move to production. The integration version to be deployed should have been successfully verified with the integration tests, otherwise we notify the risk before proceeding. After that, we change the draft status on the merge request, which enables the merge action over the production branch. We perform the merging that updates the production version. Finally, we communicate the change to the EMERALD project. Figure 4. A Merge Request mechanism to produce a new release in EMERALD 3.2.2 Public Assets Release For the public release of the CaaS Framework (https://git.code.tecnalia.dev/emerald/public), we have defined the following strategy: • A common structure for the components has been defined that is replicated in the private and public repositories. • An automated procedure has been implemented to promote the components from the private to the public repository. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 35 of 54 www.emerald-he.eu • The CaaS Framework is located at the root of the public repository. • A description of the public repository is included at the top of the public repository. 3.2.3 Keycloak configuration The strategy with regards to the identity and access management is to assume it as an external service. We follow this approach because, in a real deployment, we could be facing a situation in which we are requested to use an existing identity and access management service in which we may not have administration rights. So, the strategy for the Keycloak configuration management is reducing the administration requirements to the minimum. In this case, we will assume that in order to deploy the EMERALD CaaS framework we will only require a pre-existing realm and a user with rights to configure that realm. Using that user, during the startup of the EMERALD framework, we upload the configuration of each component to the Keycloak service. This is done through a Keycloak loading job in each of the components. This supports the following objectives: • Reduce the configuration requirements to the minimum. • Allow the components to be deployed in a pre-existing Keycloak service. • Facilitate the evolution of the Keycloak configuration of the components in an isolated way. 3.2.4 Demo pilot One of the main goals of the EMERALD methodology, apart from the creation of the CaaS Framework, is to facilitate the validation of the CaaS Framework in the pilots. With the purpose of checking the deployment of a Demo pilot and its update, a separate demo environment will be created. So, we will create a repository for the demo pilot (emerald/private/devops/demo) that will contain the necessary elements for deployment: • Configuration files for the demo pilot • Secrets for the demo pilot • Keycloak deployment for the demo pilot • CaaS Framework deployment for the demo pilot. 3.2.5 Documentation The documentation regarding the CaaS Framework is generated as part of the deployment to production. So, the focus of the documentation is on the deployment and configuration of the pilots. We provide the installation and configuration instructions for the CaaS Framework in the README.md file of the public repository, which includes information about: • Initial deployment of the CaaS Framework. Including details on how to configure the CaaS Framework to adjust to the specific needs of the pilots • Update of the CaaS Framework • Information about the feedback channels • Information about the contribution mechanisms • Information about the project roadmap • General information about Authors, Acknowledgment and License • Information about the project status D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 36 of 54 www.emerald-he.eu More contents will be added to the documentation in case it is necessary, for example to explain the usage of the evidence collector components in the pilots. Currently the documentation is available in the public repository of the EMERALD project (https://git.code.tecnalia.dev/emerald/public/caas-framework).The documentation is included in the README.md file of the public repository. Other documentation mechanisms may be added in the future to complement the installation and configuration information, such as the information generated using the pages feature of GitLab 22 . 3.2.6 Environments with IaC The production environment is deployed using IaC, with the same approach used for the integration environment. Besides, the DevOps team will support the generation of additional environments, if needed. For example, environments on pilots’ premises due to privacy or legal restrictions. The strategy for the production environment IaC is similar to the strategy with the integration IaC. That is, as any other activity in the DevOps team, it is documented in an issue related to an EMERALD project requirement and implemented through a merge request. 3.2.7 Automation The DevOps strategy works towards the automation of all activities related to the creation of releases and their deployment in the production environment. The main aspects of automation are focused on: • Deploying specific releases to specific environments. • Updating the monitoring mechanism to measure the long-term health of the CaaS Framework. The automation strategy during deployment is similar to that of integration. We use GitLab Agent for Kubernetes to translate new releases on the production environment. For the pilot environments, we leave it up to the pilot owners to decide the procedure for updating their respective environments. 22 https://docs.gitlab.com/user/project/pages/ D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 37 of 54 www.emerald-he.eu 4 Conclusions This document is the second and final version of the DevOps approach for EMERALD, where we have leveraged the baselines of the DevOps strategy presented in the previous version a year ago. In a year’s time, we have put in practice the initial ideas to release the first version of the CaaS Framework, learnt some lessons and further develop the integration approach. The result of this process is this updated version of the report, containing a detailed description of the CI/CD process implemented. The main guiding principle of this version is to provide the elements that help in the support of the transition of the EMERALD CaaS Framework to the pilots. This support will aim not only the initial deployment of the CaaS Framework, but also the evolution of the CaaS Framework during the project. The methodology used customizes the commonly used DevOps lifecycle to the characteristics and constrains of the EMERALD project. This lifecycle consists of the following steps: Plan, Code, Build, Test, Release, Deploy, Operate, and Monitor. The goals of the defined methodology are to be release-based, manage feedback, manage components, keep traceability, manage the environments, and integrate as soon as possible. In this line, we have presented the main customised processes, as well as a tailored approach to iterate, so that we prioritise speed of integration over other elements. In the CI/CD strategy part, we have described the technical approaches that we will implement to support the EMERALD project needs. In this sense, we will leverage some technologies and state of the practice DevOps resources, such as: • Configuration management with IaC • GitLab features with respect to: o Issues o Git workflows (branches and merge requests) o Automation with GitLab CI/CD Components o Documentation • Releases with containers • Container orchestration technologies It is likely that during the course of the project some changes will be made to the DevOps methodology and the CI/CD Strategy, however the fundamental processes have already been defined in this deliverable and are in operation. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 38 of 54 www.emerald-he.eu 5 References [1] EMERALD Consortium, “D1.5 DevOps methodology and CI/CD strategy for EMERALD-v1,” 2024. [2] EMERALD Consortium, “Home page,” [Online]. Available: https://www.emerald-he.eu/. [Accessed April 2025]. [3] EMERALD Consortium, “EMERALD - Annex 1 - Description of Action - GA 101120688,” 2022. [4] ISO, “ISO 16290:2013, Space systems — Definition of the Technology Readiness Levels (TRLs) and their criteria of assessment,” 2013. [5] EMERALD Consortium, “D1.3 EMERALD solution architecture-v1,” 2024. [6] EMERALD Consortium, “D1.4 EMERALD solution architecture - V2,” 2025. [7] CMMI Dev, “CMMI for Development, Version 1.3,” Software Engineering Institute (SEI), Ed., 2010. [8] International Organisation for Standardization (ISO/IEC), “ISO/IEC 15504-1:2004 Information Technology – Process Assessment – Part 1: Concepts and Vocabulary,” 2004. [9] AXELOS, “ITIL Foundation,” Stationery Office Books, Norwich, England, 2019. [10] J. A. V. M. K. Jayakody and a. W. M. J. I. Wijayanayake, “Process Improvement Framework for DevOps Adoption in Software Development,” in 2023 International Research Conference on Smart Computing and Systems Engineering (SCSE), IEEE, Jun. 2023. doi: 10.1109/scse59836.2023.10214992, 2023. [11] I. Bucena and M. Kirikova, “Simplifying the DevOps Adoption Process,” in BIR Workshops, pp. 1–15, 2017. [12] R. d. Feijter, “Towards the adoption of DevOps in software product organizations: A Maturity Model Approach,” Master’s Thesis, 2017. [13] S. Badshah, A. A. Khan and B. Khan, “Towards Process Improvement in DevOps: A Systematic Literature Review’,” in Proceedings of the Evaluation and Assessment in Software Engineering, EASE ’20. ACM, Apr. 2020. doi: 10.1145/3383219.3383280. [14] R. Amaro, R. Pereira and M. M. da Silva, “Capabilities and Practices in DevOps: A Multivocal Literature Review,” in IEEE Trans. Softw. Eng., vol. 49, no. 2, pp. 883–901, Feb. 2023, doi: 10.1109/tse.2022.3166626. [15] M. Gasparaitė and S. Ragaišis, “Comparison of devops maturity models,” in IVUS 2019. Proceedings of the International Conference on Information Technologies Kaunas, Lithuania, April 25, 2019, CEUR-WS. org, 2019, pp. 65–69. [16] R. T. Yarlagadda, “DevOps and its practices,” in Int. J. Creat. Res. Thoughts IJCRT ISSN, pp. 2320–2882, 2021. [17] A. Colantoni, L. Berardinelli and M. Wimmer, “DevopsML: Towards modeling devops processes and platforms,” in Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, pp. 1–1, 2020. [18] R. Amaro, R. Pereira and M. M. da Silva, “DevOps Metrics and KPIs: A Multivocal Literature Review,” in ACM Comput. Surv., Mar. 2024, doi: 10.1145/3652508. [19] A. V. Jha et al., “From theory to practice: Understanding DevOps culture and mindset,” in Cogent Eng., vol. 10, no. 1, p. 2251758, 2023. [20] H. R. Kadaskar, “Unleashing the Power of Devops in Software Development,” in Int. J. Sci. Res. Mod. Sci. Technol., vol. 3, no. 3, pp. 01–07, 2024. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 39 of 54 www.emerald-he.eu [21] Wikipedia, “DevOps toolchain,” 20 Jan 2024. [Online]. Available: https://en.wikipedia.org/w/index.php?title=DevOps_toolchain&oldid=1197449470. [Accessed Apr 2025]. [22] A. M. Davis, E. H. Bersoff and a. E. R. Comer, “A strategy for comparing alternative software development life cycle models,” in IEEE Trans. Softw. Eng., vol. 14, no. 10, pp. 1453–1461, doi: 10.1109/32.6190, 1988. [23] A. Mishra and D. Dubey, “A comparative study of different software development life cycle models in different scenarios,” in Int. J. Adv. Res. Comput. Sci. Manag. Stud., vol. 1, no. 5, 2013. [24] M. Kalske and others, “Transforming monolithic architecture towards microservice architecture,” Univ. Hels., 2017. [25] J.-P. Gouigoux and D. Tamzalit, “From Monolith to Microservices: Lessons Learned on an Industrial Migration to a Web Oriented Architecture,” in 2017 IEEE International Conference on Software Architecture Workshops (ICSAW), Gothenburg, France: IEEE, Apr. 2. [26] EMERALD Consortium, “D7.1 Project Manual and Quality Plan,” 2024. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 40 of 54 www.emerald-he.eu APPENDIX A: Project Risks and impact in the DevOps Methodology This section makes an analysis of some of the risks defined in the project – initially from the DoA [3] and then extended in D7.1 [26] - and the impact that the DevOps Methodology can have to mitigate them. It is important to note that these can evolve as part of the EMERALD Task 7.2 Quality Assurance & Risk Management. Table 2. Risk and mitigation list Risk n. Description Proposed Mitigation Measures 1 EUCS is not ready until 2026. Not relevant. 2 Incompatibility between OSCAL and EMERALD (data import/export, modelling of security schemes). Not relevant. 3 Users experience low usability. WP4 will work in the UI/UX concept. The methodology contains activities to manage feedback from pilots. 4 EMERALD components are not able to be fully integrated. The methodology should control which components are integrated and which components are not integrated. 5 Data set not sufficient for reaching TRL7 on the evidence collector components. Not relevant. It should be controlled by the validation activities as part of other work packages. 6 The implementation does not cover all the use cases. The methodology deployment activities should keep track of the use cases involved. 7 Underestimation of effort needed to complete activities. The methodological approach promotes short cycles that will help to identify those situations faster. Besides, short cycles will focus on having running versions, and clearer view of what is missing. 8 Technology changes require significant redesign of the EMERALD architecture. Integration tests will help to verify the redesigned elements. This will speed up the verification of refactored components as they are changed to the new architecture. 9 A partner fails to meet the obligations and becomes nonperforming or even defaulting. The methodological approach should promote multiple versions to have partial versions (instead of no versions) in that case. 10 Partner heterogeneity: The different organizational and national cultures cause collaboration problems or conflicts in the project consortium The methodology has been defined at the beginning of the project, and the clarity of the process paves the way for an easier collaboration. 13 Project execution risks: a) key milestones are delayed b) critical deliverables are delayed The DevOps methodology, bringing together the work of developers, integrators and final users, helps to mitigate the possible delays. 14 Project key technologies, development risks: The DevOps Methodology can be easily adapted to cover other development languages or technologies. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 41 of 54 www.emerald-he.eu a) Key technologies or components are not available at the expected time b) development takes longer than expected c) wrong technology base is selected d) lacking consensus on the technological approach between scientific partners The DevOps Methodology automates the integration of the source code, and thus can speed up the deployment of delayed releases to make them available for the Use Cases. 15 Use case implementation is poor The DevOps Methodology will produce three releases, as defined in the project plan, and the successive feedback can help making a better final implementation. Also, the IaC approach of the DevOps Methodology is a mitigation measure. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 48 of 54 www.emerald-he.eu name: rcm-frontend options: disableNameSuffixHash: true - envs: - ./.secrets/rcm_converter name: rcm-converter options: disableNameSuffixHash: true resources: - ./01-consul-service.yaml - ./02-consul-deployment.yaml - ./03-consul-ingress.yaml - ./04-consul-config-loader-deployment.yaml - ./05-mariadb-service.yaml - ./06-mariadb-persistance-volume-claim.yaml - ./07-mariadb-deployment.yaml - ./08-adminer-service.yaml - ./09-adminer-deployment.yaml - ./10-adminer-ingress.yaml - ./11-mariadb-setup-dbs-job.yaml - ./12-backend-service.yaml - ./13-backend-deployment.yaml - ./14-frontend-service.yaml - ./15-frontend-deployment.yaml - ./16-frontend-ingress.yaml - ./17-converter-service.yaml - ./18-converter-deployment.yaml - ./19-mariadb-setup-db-job.yaml components: - patches/keycloak-loader Listing 9. Kustomize RCM component In this Kustomize code we can see some of the common Kustomize elements used: • configMapGenerator: to load files as configmaps • secretGenerator: to load secrets of different types • resources: to add the manifests that declare the volume claims, services, deployments, ingresses, etc. to be created in Kubernetes • components: to add other components C.4 – Docker compose approach In this section we present the structure of the local development framework used for the internal test based on Docker Compose. Figure 8 presents the structure of this framework. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 49 of 54 www.emerald-he.eu Figure 8. Docker Compose framework The most remarkable elements in this structure are: • components: In the git folder we include the projects of the different EMERALD components such as RCM. They are linked using the git submodule approach. • data: In the data folder we point to the CaaS framework project with the git submodule mechanism. The usage of this framework is simple once it is defined. If we are checking a component, we start it with: docker compose up <component name> To know the components included we issue a: docker compose config --services Figure 9 shows the current list of services supported in the local development environment. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 50 of 54 www.emerald-he.eu Figure 9. Local environment services For example, if we are analysing the rcm-backend we just need to issue: docker compose up rcm-backend This will start the dependent services and once all the prerequisites have been started the rcmbackend will start, as shown in Figure 10. The advantage appears when we are testing the effect of a small change. We can perform a change in the inner services, such as changing a property in the rcm-backend code (e.g., a description, as shown in Figure 10). Figure 10. RCM change Once the change has been made, to test again we only have to issue: docker compose build rcm-backend docker compose up rcm-backend docker compose log rcm-backend C.5 – CI/CD Examples In the internal area of the GitLab repository there are several examples on how to apply the CI/CD Components. Here we describe some of them: • Docker • Semantic Versioning Figure 11 shows the details of a Docker example that includes four files: • .gitlab-ci.yml: This is the most important file that will be explained below. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 51 of 54 www.emerald-he.eu • Dockerfile: This is the main Docker artifact. We added a very simple Docker file that adds some packages to a base Ubuntu image. • LICENSE: This is a must have for any kind of repository. • README.md: This explains the repository and its purpose. Figure 11. Docker CI/CD example The focus of the example is to provide information on the GitLab CI/CD configuration. Listing 10 shows the content of the file. include: - component: git.code.tecnalia.dev/smartdatalab/public/ci-cdcomponents/docker/gitlab-ci-docker@5 inputs: snapshot-image: $CI_REGISTRY_IMAGE/snapshot:$CI_COMMIT_REF_SLUG release-image: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME kaniko-build-job-tags: ["docker"] docker-publish-job-tags: ["docker"] # https://github.com/jfrog/artifactory-userplugins/blob/master/cleanup/cleanDockerImages/README.md metadata: >- --label org.opencontainers.image.url=${CI_PROJECT_URL} --label org.opencontainers.image.source=${CI_PROJECT_URL} --label org.opencontainers.image.title=${CI_PROJECT_PATH} --label org.opencontainers.image.ref.name=${CI_COMMIT_REF_NAME} --label org.opencontainers.image.revision=${CI_COMMIT_SHA} --label org.opencontainers.image.created=${CI_JOB_STARTED_AT} --label com.jfrog.artifactory.retention.maxCount=4 prod-publish-strategy: auto hadolint-disabled: true healthcheck-disabled: true trivy-disabled: true sbom-disabled: true variables: CI_REGISTRY_IMAGE: emerald-docker-dev-local.artifact.tecnalia.dev/template-docker # CI_REGISTRY_USER defined in GitLab CI/CD settings # CI_REGISTRY_PASSWORD defined in GitLab CI/CD settings Listing 10. CI/CD for docker generation D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 52 of 54 www.emerald-he.eu In the .gitlab-ci.yml code above you can see the usage of the release 5 Docker component (https://git.code.tecnalia.dev/smartdatalab/public/ci-cd-components/docker/gitlab-cidocker@5). This means that the last release 5 will be applied. The component behaviour is controlled with input parameters, where we can: • Specify the names of the images • Filter the runners applicable for each of the jobs • Add metadata to the image before publishing into Artifactory • Select the building engine • Enable or disable additional test and features. The example is used internally, as well as the example pipelines (see Figure 12). Figure 12. docker CI/CD example pipelines We can also see the details of each stage (see Figure 13). Figure 13. Docker CI/CD stages detail Semantic Release examples follow a similar structure (see Figure 14). This example builds on the Docker example by adding semantic versioning capability. D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 53 of 54 www.emerald-he.eu Figure 14. Semantic release CI/CD example The focus of this example is to provide information on the GitLab CI/CD configuration. Listing 11 shows the content of the file. include: - component: git.code.tecnalia.dev/smartdatalab/public/ci-cdcomponents/docker/gitlab-ci-docker@5 inputs: snapshot-image: $CI_REGISTRY_IMAGE/snapshot:$CI_COMMIT_REF_SLUG release-image: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME kaniko-build-job-tags: ["docker"] docker-publish-job-tags: ["docker"] # https://github.com/jfrog/artifactory-userplugins/blob/master/cleanup/cleanDockerImages/README.md metadata: >- --label org.opencontainers.image.url=${CI_PROJECT_URL} --label org.opencontainers.image.source=${CI_PROJECT_URL} --label org.opencontainers.image.title=${CI_PROJECT_PATH} --label org.opencontainers.image.ref.name=${CI_COMMIT_REF_NAME} --label org.opencontainers.image.revision=${CI_COMMIT_SHA} --label org.opencontainers.image.created=${CI_JOB_STARTED_AT} --label com.jfrog.artifactory.retention.maxCount=4 prod-publish-strategy: auto hadolint-disabled: true healthcheck-disabled: true trivy-disabled: true sbom-disabled: true - component: git.code.tecnalia.dev/smartdatalab/public/ci-cd-components/semanticrelease/gitlab-ci-semrel@3 inputs: release-disabled: false semantic-release-job-tags: ["docker"] auto-release-enabled: true D1.6 – DevOps methodology and CI/CD strategy for EMERALD-v2 Version 1.0 – Final. Date: 30.04.2025 © EMERALD Consortium Contract No. GA 101120688 Page 54 of 54 www.emerald-he.eu branches-ref: "/^(master|main)$/" variables: CI_REGISTRY_IMAGE: emerald-docker-dev-local.artifact.tecnalia.dev/template-docker # CI_REGISTRY_USER defined in GitLab CI/CD settings # CI_REGISTRY_PASSWORD defined in GitLab CI/CD settings Listing 11. CI/CD for semantic release generation In the .gitlab-ci.yml code above you can see how we extend the previous Docker example with five lines to declare the component’s Semantic release and parametrize its behaviour. We have also tested it, and you can see the pipelines and details of their stages. In the image below (Figure 15) we can see that it is similar to the Docker pipeline but with an additional job. Figure 15. Semantic release CI/CD stages detail