FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES

FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES AUGUST 2025 AUTHOR(S): Olli Glorioso SUPERVISOR(S): Jack Munday CERN openlab Report 1/2025 PROJECT SPECIFICATION Active-Active deployments across multiple data centers are the gold standard for achieving highly available and fault resilient services. However, distributing workloads across many clusters presents significant challenges particularly for stateful deployments. This project will focus on enabling active-active kubernetes deployments across CERN’s data centers by: * Evaluating existing CNI plugin support for cross-cluster networking. * Experimenting with service mesh based solutions to provide inter-cluster traffic routing, load balancing and automated failover. * Investigating current offerings in the CNCF for multi-cluster active-active database deployments. The outcomes of this project will help establish a robust framework for active-active Kubernetes deployments, supporting business continuity and disaster recovery efforts at CERN. FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 1 CERN openlab Report 1/2025 ABSTRACT High Availability is the design of systems and services to ensure they remain operation and accessible with minimal downtime, even in the face of server failures. It is a broader field of Business Continuity, which is a set of strategies to keep business operations during and after disruptions. The four levels of Business Continuity are described in the diagram below. The levels go from Cold with hours or even days of downtime, to Active-Active with practically no downtime. In this report, I will focus on Active-Active and Active-Passive levels. Active-Active is a configuration where multiple clusters are running the same application simultaneously, sharing the load and providing redundancy. Active-Passive is a configuration where one cluster is active and serving requests, while another cluster is on standby, ready to take over in case of failure. In the context of this report, Active-Passive only provides redundancy for databases, which are continuously replicated across clusters. In the table above, the Active-Passive covers all the levels before Active-Active. FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 2 CERN openlab Report 1/2025 TABLE OF CONTENTS 1 Introduction 4 2 Active-Active Clusters for User Traffic Routing 4 2.1 InitialInstallation................................... 6 2.2 LoadBalancerSetup ................................. 7 2.3 DeployingGlobalServices .............................. 8 2.4 Verifying Cluster Mesh Connectivity . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Active-Passive Setup for Databases 11 3.1 Stateful Kubernetes Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2 CloudNativePG.................................... 12 3.2.1 Install CNPG Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2.2 Deploy Primary Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.3 Deploy Replica Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.4 Verify Replication Connection . . . . . . . . . . . . . . . . . . . . . . . . 15 3.3 Valkey......................................... 15 3.3.1 Deploy Primary Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.3.2 Deploy Replica Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.3.3 Verify Replication Connection . . . . . . . . . . . . . . . . . . . . . . . . 18 3.4 OpenSearch ...................................... 18 3.4.1 Deploy Primary Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.4.2 Deploy Replica Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4.3 Start Replication And Verify Replication Connection . . . . . . . . . . . 20 4 Conclusion 22 5 References 23 FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 3 CERN openlab Report 1/2025 1 Introduction High Availability is the design of systems and services to ensure they remain operation and accessible with minimal downtime, even in the face of server failures. It is a broader field of Business Continuity, which is a set of strategies to keep business operations during and after disruptions. The four levels of Business Continuity are described in the diagram below. The levels go from Cold with hours or even days of downtime, to Active-Active with practically no downtime. The levels of business continuity in IT are the following: Cold Pilot Light Warm Standby Active-Active RTO/RPO : hoursdays RTO/RPO : hours RTO/RPO : minuteshours RTO/RPO : zero Systems offline Minimal critical systems running Systems running at reduced capacity All systems active Restore from backup Core data live Data partially live Fully live data Restore after the event Moderate scaling after event Scale to full capacity after event Seamless failover $ $$ $$$ $$$$ In general, the three first ones (Cold, Pilot Light, Warm Standby) could be described as Active-Passive strategies, since in each one the services are ran in a somewhat reduced capacity or not at all and without any user traffic, thus making them "passive". On the other hand, there is the main server actively serving the users’ traffic. The last strategy, Active-Active, is a more complex strategy where two instances of a same service are ran simultaneously and they both are serving the users’ requests. If one of them goes down, the other instance can still continue serving the users. This report will concentrate on presenting frameworks to implement solutions for two specific categories of business continuity for multi-cluster Kubernetes scenarios: 1. Active-active clusters for user traffic routing 2. Active-passive clusters for data availability and replication Active-active clusters for user traffic routing stands for a multi-cluster solutions, where the applications are running in two separate clusters actively, and the user traffic is routed to both of them. Applications are usually Kubernetes pods or deployments, so to copy them one can just copy the manifests and apply them in the other cluster. Active-passive clusters for data availability and replication stands for multi-cluster solutions, where there is a primary cluster running the actual application with a read/write database and secondary cluster running a read-only database and possible the actual application in a standby mode. The user traffic is routed to the primary cluster and the database in the secondary cluster replicates the primary cluster’s database in real-time. In case of a failure of the primary cluster, the secondary cluster can be promoted to primary and can start serving the users reasonably fast, since the original data is already there. 2 Active-Active Clusters for User Traffic Routing Cilium is a Kubernetes CNI (Container Network Interface) plugin that provides advanced networking and security features for Kubernetes clusters. It is designed to enhance the performance, scalability, and security of containerized applications. Cilium uses eBPF (extended FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 4 CERN openlab Report 1/2025 Figure 1: Target Cilium Cluster Mesh architecture with two clusters. Berkeley Packet Filter) technology to implement networking and security policies at the kernel level, allowing for efficient and flexible network management. Cilium Cluster Mesh is a feature that allows multiple Kubernetes clusters to be connected and managed as a single logical network [1]. This enables seamless communication between pods across different clusters. Cluster Mesh is particularly useful for multi-cluster deployments, where applications need to span multiple clusters for high availability. This feature differs from service mesh, which is a layer that provides communication between services within a cluster, whereas Cluster Mesh focuses on communication between clusters. The benefit of Cilium Cluster Mesh is that it allows for seamless communication between pods across different clusters, enabling load balancing and failover capabilities. It allows user traffic to be distributed across multiple clusters, ensuring that if one cluster fails, the other clusters can continue to serve requests. Furthermore, with Cilium Cluster Mesh it is seamless to label services as global, which allows them to be discovered and accessed from any cluster in the mesh [3]. This way it is also easy to group pods together, as the pods with the same names in different groups perform the load balancing just between each other. This chapter will cover the setup of Cilium Cluster Mesh for Active-Active user traffic. The setup involves configuring multiple pods in different clusters, and then load balancing the traffic across these pods. The goal is to ensure that user requests are distributed evenly across the clusters, providing redundancy and high availability even if one of the clusters fail. The diagram below illustrates the architecture used for this setup, with the APIand ML-services running in different clusters, and the user traffic being load balanced across them pairwise. FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 5 CERN openlab Report 1/2025 2.1 Initial Installation Installing Cilium and Cilium Cluster Mesh is straightforward with the Cilium CLI, and you can follow this guide by Cilium to get it installed. However, users may encounter issues with the stallation via Cilium CLI, especially if they have a larger umbrella Helm chart for all their installations. Furthermore, when running the ‘cilium clustermesh connect‘ command on the CLI installation, the Cilium installation exceeded Helm release size limit of 1 MB. To overcome this, one can install Cilium Cluster Mesh manually with Helm. Let’s assume that one has two clusters named cilium-001 and cilium-002, both with cernmanager installed. On a high level, it can be done as follows: 1. Create the Kubernetes clusters and install Cilium in them via Helm. # Run this against both of the Kubernetes clusters. helm repo add cilium https://helm.cilium.io/ helm install -n kube-system cilium cilium/cilium --create-namespace \ --set clustermesh.useAPIServer=true --version 1.18.0 2. Use the following YAML configuration to upgrade the Helm installation after you have the clusters’ master node names with ‘kubectl get no‘, api-server secrets with ‘kubectl get secret -n kube-system clustermesh-apiserver-remote-cert -o jsonpath=’.data’ | jq .‘ and master node internal IP addresses with ‘kubectl get no -owide‘. The example is for cilium-002 cluster, it is the same for cilium-001 but with opposing values. --- # cilium-002.yaml cilium: cluster: # Master node name from kubectl get no name: <CILIUM-002-MASTER-NODE-NAME> # Cluster ID, can be any number, but should be unique across clusters. id: 002 # Mandatory to fix issue mentioned in # https://github.com/cilium/cilium/issues/20942 bpf: masquerade: true clustermesh: useAPIServer: true config: enabled: true clusters: # Second cluster master node name from kubectl get no. -name: <CILIUM-001-MASTER-NODE-NAME> port: 32379 # Second cluster internal IP address from kubectl get no -owide. ips: - <CILIUM-001-MASTER-NODE-IP> # b64 contents from last step. Yes, you have to hardcode them here tls: key: <APISERVER-KEY> FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 6 CERN openlab Report 1/2025 cert: <APISERVER-CERT> caCert: <APISERVER-CA-CERT> 2.2 Load Balancer Setup To enable external access to the cluster with this integration, an ingress must be deployed, which in turn automatically provisions a load balancer for the cluster. Ingress-nginx ingress controller is used for this purpose, as with Cilium ingress controller [2] I encountered problems with the cluster networking (see more in the Troubleshooting section). Install the ingress-nginx controller with the following Helm configuration: # cilium-002-ingress-controller.yaml ingress-nginx: controller: nodeSelector: role: ingress service: enabled: true nodePorts: http:"" https:"" type: LoadBalancer enabled: true Listing 1: Configuration to enable ingress-nginx. Since the Helm configuration deployed the load balancer into a node with label *ingress*, we should label one as such, preferably before the ingress-nginx installation: kubectl label node <NODE-OF-CHOICE> role=ingress Next up, we should deploy the ingress and thus the load balancer by applying this manifest: # ingress-manifest.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: sample-http-ingress annotations: # This annotation added to get the setup to work # Read more at https://github.com/cilium/cilium/issues/ # 25818#issuecomment-1572037533 # and at https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/ # nginx-configuration/annotations.md#service-upstream nginx.ingress.kubernetes.io/service-upstream:"true" spec: ingressClassName: nginx rules: -host: FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 7 CERN openlab Report 1/2025 http: paths: -path: / pathType: Prefix backend: service: # Name of a global service backend. # If you would deploy another service, you would change the name. name: api-service port: number: 8080 Listing 2: Configuration to deploy ingress-nginx load balancer. Deploy this manifest with kubectl apply -f ingress-manifest.yaml. The load balancer will be provisioned automatically, and the ingress controller will start routing traffic to the specified backend service later. Apply this in both clusters. Then, configure DNS load balancing by assigning the same DNS name to the external addresses of both load balancers. This way, when clients resolve the DNS name, the DNS service distributes requests across the available load balancers. This step depends on the DNS service you are using. After deploying the ingress in both clusters, one should have two load balancers deployed in OpenStack. To deploy an abstract umbrella load balancer that balances between these two clusters, one has to give them aliases with same DNS names, and LanDB will take care of the rest. With other DNS services setting this up is service-specific. openstack loadbalancer set --tag landb-alias=<DNS_NAME>--load-1openstack loadbalancer set --tag landb-alias=<DNS_NAME>--load-22.3 Deploying Global Services The automatic load-balancing between clusters can be achieved by defining Kubernetes ClusterIPservices with identical names and namespaces and by adding the annotation service.cilium.io/global: "true" to declare them global. Cilium will take care of the rest. Furthermore, since this guide is utilizing an external ingress controller, an additional annotation is needed for the global services, namely service.cilium.io/global-sync-endpoint-slices: "true". Apply the following CRD in both of the clusters to create the global service ClusterIP, and mock pods within them: # global-api-service-manifest.yaml apiVersion: v1 kind: Service metadata: # The name and namespace need to be the same across services in different clusters. # This name is important as it defines the load balancing groups for Cilium. name: api-service annotations: # Declare the global service. # Read more here: https://docs.cilium.io/en/stable/network/clustermesh/services/ service.cilium.io/global:"true" # Allow the service discovery with third-party ingress controllers. FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 8 CERN openlab Report 1/2025 3.2.4 Verify Replication Connection In order to confirm that the replication is actually working, one has to log in to the primary cluster, insert some test data, then log in to replica cluster and verify that the data appears there as well. You can do that by following these steps: 1. Get the primary cluster secret in cnpg-primary: kubectl get secret cluster-example-app -n \ cnpg-system -o jsonpath="{.data.password}" | base64 -d 2. Create a debug pod in cnpg-primary: kubectl run psql-client --rm -it --restart=Never --image=postgres -- bash 3. Access the database (run this inside the pod) and insert the password in cnpg-primary: psql -h cluster-example-rw.cnpg-system.svc -U app -d app 4. Create a table and add a row in cnpg-primary: CREATE TABLE my_table ( name VARCHAR(100) ); INSERT INTO my_table (name) VALUES ('DOG'); Once the primary cluster is all set up, you can log in to the replica cluster in the same way and check that the data has been replicated: 1. Get the replica cluster username and password from THE PRIMARY CLUSTER (replica cluster uses the same roles as the primary). Thus, you can reuse the password from the previous section. 2. Create a debug pod in the replica cluster in cnpg-replica: kubectl run psql-client --rm -it --restart=Never --image=postgres -- bash 3. Access the database (run inside pod) and insert the password in cnpg-replica: psql -h cluster-replica-example-rw.cnpg-system.svc -U app -d app 4. Check that the same test rows exist in the replica cluster in cnpg-replica: SELECT *FROM my_table; You may also use CloudNativePG kubectl plugin and use some of the commands below [5]. # Run this in both and check for errors and that the LSN state matches. kubectl cnpg status <CNPG-CLUSTER-NAME> --verbose 3.3 Valkey For Valkey, the supported replication is called leader follower (primary-replica) [6]. The replica will automatically reconnect to the primary every time the replication link breaks, and will always attempt to be an exact copy of it regardless what is happening. FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 15 CERN openlab Report 1/2025 3.3.1 Deploy Primary Instance It is straightforward to install Valkey with the Helm chart provided by Bitnami. Copy the following configuration to a file called valkey-primary.yaml. # valkey-primary.yaml global:# Storage class of your choice. storageClass: <STORAGE-CLASS> replica: replicaCount: 0 primary: replicaCount: 1 persistence: size: <PERSISTENCE-SIZE> # Choose a storage size, e.g. 1Gi. sizeLimit: <PERSISTENCE-SIZE-LIMIT> # Choose a storage size limit, e.g. 1Gi. service: # Deploys a load balancer for you. type: LoadBalancer port: 6380 auth: enabled: true password: <PRIMARY-PASSWORD> # You can also use Valkey-generated password. tls: enabled: true autoGenerated: false # True if you want to get generated keys. existingSecret: myvalkey-crt # Name of the secret containing the TLS certificates. certFilename: tls.crt certKeyFilename: tls.key certCAFilename: ca.crt authClients: true Listing 6: Custom Helm values for Valkey primary instance. This configuration will deploy a primary Valkey instance with a load balancer, persistence, and authentication enabled. The TLS is also enabled, but you need to provide your own certificate secret myvalkey-crt in the same namespace as the Valkey instance. You can create this secret with the following command: # For this, you would need to create client.crt, client.key and ca.crt # files with OpenSSL or similar tool. # If you set autoGenerated: true, this secret will be created automatically. kubectl create secret generic myvalkey-crt \ --from-file=tls.crt=path/to/client.crt \ --from-file=tls.key=path/to/client.key \ --from-file=ca.crt=path/to/ca.crt Then, you can install Valkey with the given yaml configuration by running: helm install myvalkey \ oci://registry.cern.ch/docker.io/bitnamicharts/valkey \ -n valkey -f valkey-primary.yaml --create-namespace \ --version 3.0.30 FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 16 CERN openlab Report 1/2025 3.3.2 Deploy Replica Instance Next up – deploying the replica. Custom Helm configuration for it: global: defaultStorageClass: <STORAGE-CLASS> storageClass: <STORAGE-CLASS> replica: replicaCount: 1 # Master database load balancer IP address, and password. configuration: |- replicaof <MASTER-IP> 6380 masterauth <MASTER-PASSWORD> tls-replication yes persistence: size: <PERSISTENCE-SIZE> # Choose a storage size, e.g. 1Gi. sizeLimit: <PERSISTENCE-SIZE-LIMIT> # Choose a storage size limit, e.g. 1Gi. readinessProbe:# Breaks if these not set enabled: false livenessProbe: enabled: false auth: enabled: true password: <REPLICA-PASSWORD> # Password of your choice for the replica. primary: replicaCount: 0 # No primary databases persistence: size: <PERSISTENCE-SIZE> # Choose a storage size, e.g. 1Gi. sizeLimit: <PERSISTENCE-SIZE-LIMIT> # Choose a storage size limit, e.g. 1Gi. tls: enabled: true autoGenerated: false # For replica this needs to be false since we # are using the existing secret from primary. existingSecret: myvalkey-crt certFilename: tls.crt certKeyFilename: tls.key certCAFilename: ca.crt authClients: true Listing 7: Custom Helm values for Valkey primary instance. Similarly, you should create the myvalkey-crt secret containing the same secrets as in the primary, whether they were autogenerated or not. Look above on how to create the secret from existing files. Again, install the replica with the Helm chart: helm install myvalkey \ oci://registry.cern.ch/docker.io/bitnamicharts/valkey \ -n valkey -f valkey-replica.yaml --create-namespace \ FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 17 CERN openlab Report 1/2025 --version 3.0.30 Now you should have two Kubernetes clusters, both with a Valkey deployment running. 3.3.3 Verify Replication Connection Next, we are going to test that the replication is really working as expected. This can be done by first starting and logging to valkey-clients in both clusters with kubectl run --namespace valkey valkey-client --restart='Never'\ --env VALKEY_PASSWORD=<PRIMARY-OR-REPLICA-PASSWORD> \ --image docker.io/bitnami/valkey:8.1.3-debian-12-r1 \ --command -- sleep infinity kubectl exec --tty -i valkey-client \ --namespace valkey -- bash and then by first creating some sample data in primary instance with REDISCLI_AUTH="<PRIMARY-PASSWORD>" valkey-cli -h myvalkey-primary SET test something and finally by verifying that this sample data appears in the replica instance with REDISCLI_AUTH="<REPLICA-PASSWORD>" valkey-cli -h myvalkey-replicas GET test # Expected output # something 3.4 OpenSearch OpenSearch is an open-source search and analytics engine, forked from ElasticSearch after Elastic changed its license. It provides full-text search, log analytics and real-time monitoring. It includes a dashboard services for visualization and supports use cases like observability, security analytics and application search. 3.4.1 Deploy Primary Instance Similarly, OpenSearch will be installed with the project’s official Helm charts, by using the custom values-file below. OpenSearch will be first installed without network.publish_hostvariable, after which having the external load balancer deployed, the installation will be upgraded with the variable set to the load balancer IP. # opensearch-primary.yaml replicas: 1 extraEnvs: -name: OPENSEARCH_INITIAL_ADMIN_PASSWORD value: <PRIMARY-CLUSTER-PASSWORD> # Password of choice for the primary cluster. # Uncomment the next lines to set the primary cluster load balancer IP. # - name: "network.publish_host" FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 18 CERN openlab Report 1/2025 # value: "<PRIMARY-LB-EXTERNAL-IP>" resources: requests: memory: <PERSISTENCE-SIZE> # Choose a memory size, e.g. 1Gi. cpu: <CPU-REQUEST> # Choose a CPU request, e.g. "100m". persistence: enabled: true storageClass: <STORAGE-CLASS-NAME> # Storage class of your choice. size: <PERSISTENCE-SIZE> # Choose a storage size, e.g. 1Gi. service: type: LoadBalancer config: # This disables the security plugin, which is not needed for this setup. # It is recommended to enable it in production. opensearch.yml: | plugins.security.disabled: true Listing 8: OpenSearch primary cluster configuration. Install this with the following: helm repo add opensearch https://opensearch-project.github.io/helm-charts/ helm install myopensearch \ -n opensearch opensearch/opensearch \ -f opensearch-primary.yaml \ --version 3.2.0 Then uncomment the lines in the configuration file, which set the network.publish_host variable to the external IP of the load balancer. You can find the external IP by running: kubectl get svc -n opensearch myopensearch-opensearch This will give you the external IP of the OpenSearch load balancer. You can also find it in the OpenStack load balancer list. Once you have the external IP, you can set it in the configuration file and upgrade the installation with Helm upgrade: helm upgrade myopensearch \ -n opensearch opensearch/opensearch \ -f opensearch-primary.yaml \ --version 3.2.0 3.4.2 Deploy Replica Instance To install the replica database in the other cluster, install it with this custom values-file: replicas: 1 extraEnvs: -name: OPENSEARCH_INITIAL_ADMIN_PASSWORD value: <REPLICA-CLUSTER-PASSWORD> resources: requests: FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 19 CERN openlab Report 1/2025 memory: <PERSISTENCE-SIZE> # Choose a memory size, e.g. 1Gi. cpu: <CPU-REQUEST> # Choose a CPU request, e.g. "100m". persistence: enabled: true storageClass: <STORAGE-CLASS-NAME> size: <PERSISTENCE-SIZE> # Choose a storage size, e.g. 1Gi. config: opensearch.yml: | # Security disabled for the PoC, should be enabled in production. plugins.security.disabled: true Listing 9: OpenSearch replica cluster configuration. 3.4.3 Start Replication And Verify Replication Connection Now that both clusters have been initialized, we can set up the replication. The replication is done by using the API of OpenSearch. We can first start by creating an entry to the primary database: kubectl run curl --rm -i --tty \ --image=curlimages/curl:8.8.0 \ -- sh # Insert mys-imple-index to the primary cluster curl -X PUT "http://<PRIMARY-LB-IP>:9200/my-simple-index" # Insert a document to the primary cluster curl -X POST "http://<PRIMARY-LB-IP>:9200/my-simple-index/_doc/1" \ -H 'Content-Type: application/json'\ -d '{"message": "Hello OpenSearch"}' We can turn the replication on by making calls to the OpenSearch API in the replica database. kubectl run curl --rm -i --tty \ --image=curlimages/curl:8.8.0 \ -- sh # Start the replication curl -X PUT "http://<REPLICA-IP>:9200/_cluster/settings" \ -H 'Content-Type: application/json'-d ' { "persistent": { "cluster": { "remote": { "follower": { "seeds": ["<PRIMARY-LB-IP>:9300"] } } } } FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 20 CERN openlab Report 1/2025 } ' # Insert the same index as in the primary cluster curl -X PUT -H 'Content-Type: application/json'\ 'http://<REPLICA-IP>:9200/_plugins/_replication/follower-01/_start?pretty'\ -d ' { "leader_alias": "follower", "leader_index": "my-simple-index" }' This will start the replication from the primary cluster to the replica cluster. You can check the status of the replication by these commands: # Check the replication status curl -X GET -k \ http://<REPLICA-IP>:9200/_plugins/_replication/follower-01/_status?pretty # Expected output # { # "follower_index_status": [ # { # "follower_index": "follower-01", # "leader_alias": "remote-cluster", # "leader_index": "leader-01", # "status": "ACTIVE", # "syncing_details": { # "operations_received": 154327, # "operations_applied": 154327, # "failed_read_requests": 0, # "failed_write_requests": 0, # "last_requested_seq_no": 154327, # "last_applied_seq_no": 154327, # "behind_operations": 0, # "last_updated_timestamp": "2025-08-20T14:30:00Z" # }, # "shard_replication_status": [ # { # "shard_id": 0, # "status": "ACTIVE", # "leader_checkpoint": 51432, # "follower_checkpoint": 51432, # "seq_no_behind": 0 # }, # { # "shard_id": 1, # "status": "ACTIVE", # "leader_checkpoint": 102895, # "follower_checkpoint": 102895, FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 21 CERN openlab Report 1/2025 # "seq_no_behind": 0 # } # ] # } # ] # } # Check the document with ID 1 in the replica cluster index curl -X GET http://<REPLICA-IP>:9200/follower/_doc/1 # Expected output # { # "_index": "follower", # "_type": "_doc", # "_id": "1", # "_version": 1, # "_seq_no": 0, # "_primary_term": 1, # "found": true, # "_source": { # "message": "Hello OpenSearch" # } # } 4 Conclusion In this report, we have covered the implementation of fault tolerance solutions for multi-cluster Kubernetes deployments using Cilium Cluster Mesh and various database technologies. The setup ensures high availability and redundancy for user traffic and databases across multiple clusters. Check below the Helm chart versions of the different installations. Name Chart Version valkey 3.0.30 opensearch 3.2.0 cnpg 0.26.0 cilium 1.18.0 Table 1: Installed Helm Charts for Valkey, OpenSearch, and CNPG Future questions for this initiative include: - How to automate the failover process in case of a cluster failure, especially with the databases since they do not support it out of the box? - How to use monitoring and dashboards to visualize the health and performance of the multicluster setup? - Are there any other database technologies that would be possible to replicate in a similar manner? - How does the Cilium Cluster Mesh load balancing actually work, randomly or deterministically? How to control the load balancing behavior? We hope that this report has provided you with a comprehensive understanding of how to implement fault tolerance solutions for multi-cluster Kubernetes deployments. If you have any questions or comments, feel free to reach out! Special thanks to my supervisor, Jack, for your guidance and support throughout the project. FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 22 CERN openlab Report 1/2025 5 References [1] Cilium. Cluster Mesh. Accessed: 2025-07-28. 2025. url:https://docs.cilium.io/en/ stable/network/clustermesh/clustermesh/ (visited on 07/28/2025). [2] Cilium. Kubernetes Ingress Support.url:https : / / docs . cilium . io / en / stable / network/servicemesh/ingress/ (visited on 07/28/2025). [3] Cilium. Load-balancing Service Discovery. 2025. url:https://docs.cilium.io/en/ stable / network / clustermesh / services / #gs - clustermesh - services (visited on 07/28/2025). [4] CloudNativePG. CloudNativePG Helm Charts.url:https://github.com/cloudnativepg/charts (visited on 07/29/2025). [5] CloudNativePG. CloudNativePG Plugin.url:https://cloudnative-pg.io/documentation/ 1.20/kubectl-plugin/ (visited on 07/29/2025). [6] Valkey. Replication.url:https://valkey.io/topics/replication/ (visited on 08/04/2025). FAULT TOLERANCE SOLUTIONS FOR MULTI-CLUSTER KUBERNETES 23