UNIFIED TLS CERTIFICATE AUTOMATION USING LET'S ENCRYPT

UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT July 2025 AUTHOR: Amélie Leconte Association MiNET, Télécom SudParis SUPERVISORS: Antonio Nappi, Artur Wiecek CERN openlab Report PROJECT SPECIFICATION Keywords: TLS, HTTPS, automation, encryption, ACME protocol, HAProxy, DNS API, cybersecurity Objective: The aim of this report is to implement and evaluate an automated system for managing TLS certificates using the ACME protocol with acme.sh client. This solution is expected to simplify certificate issuance and renewal in a scalable and secure way, adaptable to CERN’s networking environment. Scope: •Research on encryption practices and TLS architecture. •Investigation of ACME protocol and the acme.sh client. •Development of a basic DNS API for domain validation. •Integration of automated certificates into a HAProxy-based and Apache-based reverse proxy with Server Name Indication. •Testing across single-node and proxy-based environments. •Evaluation of the implementation within CERN’s infrastructure constraints. Deliverables: •A functional PoC using acme.sh. •Configuration examples for DNS, proxy, and certificates. •Integration guide tailored for CERN environments. •Analysis of security benefits and operational limitations. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 1 CERN openlab Report ABSTRACT This report presents a study and proof of concept (PoC) for the automation of TLS certificate management using the ACME protocol, within the context of CERN’s infrastructure. It begins with a wide review of key concepts in cybersecurity, particularly encryption, and tackles the growing need for automating certificate issuance and renewal. The PoC includes the development of a custom DNS API, and automated certificate deployment scenarios, and the integration of HAProxy with Server Name Indication (SNI) for HTTPS termination. Finally, the report evaluates the feasibility of integrating this automated approach within CERN’s existing systems, outlining both the benefits and limitations of using acme.sh in production environments. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 2 CERN openlab Report TABLE OF CONTENTS 1 INTRODUCTION 5 2 Background Research 6 2.1 About cyber-security : Encryption . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.1 Why do we have to encrypt the data ? . . . . . . . . . . . . . . . . . . . 6 2.1.2 How to encrypt the data ? . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 About automation : Automation in TLS . . . . . . . . . . . . . . . . . . . . . . 9 2.2.1 What could we automate ? . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.2 ACMEprotocol................................ 9 3 The Proof Of Concept 12 3.1 CodingaDNSAPI.................................. 12 3.1.1 Why?..................................... 12 3.1.2 How? ..................................... 14 3.2 POC on a single DN, without proxy . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.3 POC three DNs with proxy+SNI . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.3.1 Proxymodes ................................. 21 3.3.2 Proxy configuration with SNI . . . . . . . . . . . . . . . . . . . . . . . . 22 3.3.3 Backendservers................................ 24 3.3.4 Finaltest ................................... 24 4 Integration on CERN infrastructure 28 4.1 ApacheProxy..................................... 28 4.1.1 Configuration................................. 28 4.1.2 HTMLpages ................................. 29 4.1.3 Some useful CLI command I had to use. . . . . . . . . . . . . . . . . . . 29 4.2 Kubernetesenvironment ............................... 31 4.2.1 Roadmap ................................... 31 4.2.2 Configurationfiles .............................. 32 4.2.3 Results .................................... 36 4.2.4 DebugcommandsIused........................... 36 5 Conclusion 37 5.1 Prosacme.sh ..................................... 37 5.2 Limitsofacme.sh................................... 37 5.2.1 TXT record not empty on challenge subdomain . . . . . . . . . . . . . . 37 5.2.2 Openssl showcert with SNI disabled. . . . . . . . . . . . . . . . . . . . . 37 5.3 Whatisnext? .................................... 40 5.3.1 Moreautomation............................... 40 5.3.2 About Wildcard certificates . . . . . . . . . . . . . . . . . . . . . . . . . 40 6 Appendix 41 6.1 Otherworks...................................... 41 6.1.1 Final acme.sh command used for the POC . . . . . . . . . . . . . . . . . 41 6.1.2 Updating the change-dns.py code . . . . . . . . . . . . . . . . . . . . . . 42 6.2 Usefultools ...................................... 42 6.2.1 ProxyComparison .............................. 43 UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 3 CERN openlab Report 6.2.2 CreateaACMEaccount........................... 44 6.3 Documentation .................................... 44 6.3.1 Mainlinks................................... 44 6.3.2 DebugRessources............................... 44 6.3.3 Otherlinks .................................. 45 UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 4 CERN openlab Report 1 INTRODUCTION Nowadays, a wide range of open-source tools allows anyone with basic knowledge to intercept and inspect data flowing through the Internet. Without proper encryption, sensitive information-such as authentication credentials or personal data-can easily be exposed to malicious actors. This paves the way to serious threats regarding the integrity and confidentiality of web applications. To protect against such vulnerabilities, it is essential to use SSL/TLS (Secure Sockets Layer / Transport Layer Security) protocols, which encrypt data exchanged between clients and servers. These two protocols are more commonly known to enable https instead of http when surfing a website. The use of SSL/TLS requires the maintainer of the website to obtain a digital certificate, issued by a trusted Certificate Authority (CA). Currently, at CERN, TLS certificate management for these applications involves several manual steps. This report introduces a unified solution to automate certificate management across diverse hosting environments at CERN. By leveraging Let’s Encrypt -CA used at CERNover acme.sh -automation tool for Let’s Encrypt- , and implementing Server Name Indication (SNI) in the reverse proxy infrastructure, this work enables dynamic and secure handling of certificates on a per-host basis. This paper details the architecture, implementation, and benefits of this solution, which enhances the security, and scalability of the CERN Application Hosting platform. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 5 CERN openlab Report 2 Background Research This chapter provides some background research on the project, especially the networking and cyber security background necessary to tackle this issue. 2.1 About cyber-security : Encryption 2.1.1 Why do we have to encrypt the data ? In today’s internet-driven world, most user interactions -such as browsing websites, logging into applications, or accessing online servicesinvolve the exchange of data over public or semipublic networks. These interactions are made possible through the HTTP protocol, which defines how clients and servers communicate over the web. However, while HTTP defines how data is exchanged, it does not address how this data should be secured during transmission. Without encryption, this data travels in plain text, making it vulnerable to interception by anyone with access to the network path. And for instance, tools like Wireshark allow even non-experts to capture and analyze network traffic with ease, revealing sensitive information such as login credentials, session cookies, which would be sent somehow in a webpage. This is where encryption plays a critical role. It ensures that the content of the communication cannot be understood by unauthorized entities, even if the data is intercepted in transit. Without encryption, attackers could impersonate legitimate websites (phishing), steal authentication data, or manipulate web content without the user’s knowledge. Figure 1: Encryption [SSL2BUY] The importance of encrypting web traffic is now widely recognized, and every web user is able to know if one site is secured thanks to encryption by just looking at the web search bar. If ’HTTPS’ is written, it means that the secure (encrypted !) version of HTTP is running, over SSL/TLS. Figure 2: Example of https with CERN main website UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 6 CERN openlab Report 2.1.2 How to encrypt the data ? Encrypting web traffic is achieved using one of the two following protocols: SSL or TLS, which secure the connection between a client—typically a web browser—and a server. TLS (Transport Layer Security) being the evolved and more secure version of SSL (Secure Sockets Layer). Let’s try to understand how these protocols enable encryption. These protocols relies on two tools : Cryptographic keys and Certificates. Cryptographic keys Let’s admit that a client wants to send data to a server (for instance a CERN administrator who would want to log to a CERN website by sending its credentials online). The client needs to encrypt the data sent in a way the server -only the serverwould be able to decrypt it. This is done by using cryptographic keys. Cryptographic keys come in pairs: •A public key : accessible to anyone, in particular the client ; •A private key : only known by the server. Both keys are stored in the server, but only the public key is accessible to anyone, in particular the client. The client will use this public key to encrypt the data before sending it. And because the public key was generated by the server, the server knows how to decrypt the data thanks to his private key. This is why the private key must be only known by the server, otherwise, the secret is lost ! Figure 3: Encryption between client and server [ClickSSL] Thanks to encryption, we can make sure that no one is able to understand the encrypted data traveling. But one breach still remains: how can we be sure that the server we are communicating with is actually the right server, and not an impostor pretending to be it? It would be easy for an impostor to share a public key linked to its own private key, and wait to collect the data it could decrypt. This is where digital certificates come into play. Certificates A certificate is an electronic document issued by a trusted Certificate Authority (CA), which check that a given public key truly belongs to a specific domain name (e.g., cern.ch). In other words, it serves to cryptographically bind a public key to a verified identity. When a client (like a CERN administrator) connects to a website, the server sends its certificate to the client. This certificate contains: UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 7 CERN openlab Report •the server’s public key ; •the domain name it is associated with ; •the digital signature of the Certificate Authority, which guarantees the certificate’s authenticity. If a server owns a certificate, the client can now send its data, which he will encrypt thanks to a public key provided by the server. Cryptographic keys + Certificate : TLS protocol Let’s now put together Cryptographic keys and Certificates to understand how TLS works. When a user visits a secure website (using HTTPS), their browser and the server perform a TLS handshake. During this handshake, the server presents its certificate, and the browser checks whether it has been signed by a trusted CA. If the certificate is valid and trusted, the browser proceeds to establish an encrypted session : especially sharing the server’s public key to the client to encrypt the data. Figure 4: TLS Handshake [Encryption Consulting] UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 8 CERN openlab Report _debug "nsupdate command sent" return 0 } # Called by acme.sh to remove the TXT record dns_myapi_rm() { fulldomain="$1" txtvalue="$2" _debug "Removing TXT record for domain: $fulldomain" _debug "TXT value: $txtvalue" if ! _get_root "$fulldomain";then _err "Cannot determine root zone for $fulldomain" return 1 fi subdomain="_acme-challenge" ( echo "server <IP-of-servor>" echo "zone $ZONE" echo update delete ${subdomain}.${ZONE}60 IN TXT "$txtvalue" echo "send" echo "quit" )| /usr/bin/nsupdate -y "$MYAPI_Username:$MYAPI_Password" return 0 } # txt challenge made for the subdomain _acme-challenge.db-lb-k8s-321.cern.ch, # for the root domain db-lb-k8s-321.cern.ch _get_root() { domain=$1 while true; do if host -t ns "$domain">/dev/null 2>&1;then ZONE=$domain return 0 fi domain="${domain#*.}" if [[ -z "$domain"]];then return 1 UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 15 CERN openlab Report fi _debug "Determined zone: $ZONE" done } ########################################################## # LAUNCH WITH acme.sh --issue --dns dns_myapi -d <CNAME-of-domain> # --challenge-alias <real-DN> --debug --server letsencrypt #ex : acme.sh --issue --dns dns_myapi -d amelie-letsencrypt-test.cern.ch # --challenge-alias db-lb-k8s-321.cern.ch --debug --server letsencrypt #def : fulldomain="_acme-challenge.db-lb-k8s-321.cern.ch", # ZONE=db-lb-k8s-321.cern.ch, subdomain="_acme-challenge" #The file must be placed in acme.sh/ folder, or in acme.sh/dnsapi/ subfolder The API got three functions : •dns_myapi_add() : function to add a TXT record on the DNS, based on nsupdate command. •dns_myapi_rm() : function to remove a TXT record on the DNS, based on nsupdate command. •_get_root() : function called before using nsupdate in dns_myapi_add() and dns_myapi_rm(), to identify the zone and the root of the domain from the input acme.sh command. What is important to note in this script is that the TXT record is not added to the DN directly during the DNS-01 challenge, but it is added to "_acme-challenge.<DN>" Let’s Encrypt will indeed perform the verification on that specific label, dedicated to the DNS-01 challenge. 3.2 POC on a single DN, without proxy It is now time to put in a nutshell this topic in a proof of concept. First issuing a certificate for a single DN, then for three DN, all managed by a Proxy+SNI. Automatizing on a cern.ch subdomain The proof of concept for a single VM is simple : •A Linux (Ubuntu) VM on which is downloaded the acme.sh script ; •A DN linked with the VM IP : "amelie-letsencrypt-test.cern.ch" As we said, these two things set up, one could just launch the acme command to get a certificate. But at CERN, things are different. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 16 CERN openlab Report Automatizing on a cern.ch subdomain with CNAME Let’s now dive into the subtleties of CERN infrastructure. If all we previously explained is correct, some more details may count in order to make the acme command work. The main subtility is the use of CNAME at CERN. As you can see in the following screenshot, amelie-lets-encrypt-test.cern.ch is, in reality, a CNAME for db-lb-k8s-321.cern.ch. Figure 7: CNAMEs at CERN This introduces an important issue: ACME script will follow the CNAME chain when adding the TXT record. As a result, the TXT record will not be placed under _acme-challenge.amelieletsencrypt-test.cern.ch directly, but under _acme-challenge.db-lb-k8s-321.cern.ch, the target of the CNAME. Or Let’s Encrypt, in order to provide a certificate for amelie-lets-encrypt.cern.ch will only consider the txt records stored in _acme-challenge.amelie-lets-encrypt.cern.ch, which do not exist. NB: If you try to launch the command directly for the db-lb-k8s-321.cern.ch, you will get a certificate for this DN, which is not what we want neither. Let’s admit that we tried to launch the following command : acme.sh --issue --dns dns_myapi -d <CNAME-of-domain> --debug --server letsencrypt The following diagram shows the issue encountered, at the TXT record adding step. To avoid any confusion between CNAME and DN, I left the DN and CNAME I used during my tests : •DN : db-lb-k8s-321.cern.ch ; •CNAME : amelie-letsencrypt-test.cern.ch. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 17 CERN openlab Report Figure 8: CNAME issue with ACME As depicted in this graph the first and clearest solution is to add an additionnal CNAME _acme-challenge.<CNAME-of-domain> => _acme-challenge.<real-DN> Another solution, could also obviously be to get rid of CNAME. These are compromises to discuss with stakeholders in the CERN networking infrastructure. For experiment purpose, we tried to add a CNAME between the two challenge subdomains. It worked and we managed to get a certificate for the CNAME. Note that the final lauching command is : acme.sh –issue –dns dns_myapi -d <CNAME-of-domain> –challenge-alias <realDN> –debug –server letsencrypt UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 18 CERN openlab Report Figure 9: Cert success The following diagram is the updated one with the additional CNAME _acme-challenge.<CNAME-of-domain> => _acme-challenge.<real-DN> : UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 19 CERN openlab Report Figure 10: Successful ACME protocol with CNAME UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 20 CERN openlab Report 3.3 POC three DNs with proxy+SNI Let’s now understand how certificate management can get along with Proxy and SNI. Let’s first define what these two concepts are: •Proxy: A proxy is an intermediary server that sits between a client and a destination server. It forwards client requests to the destination server and can modify, inspect, or filter traffic if needed. •SNI (Server Name Indication): SNI is an extension to the TLS protocol that allows a client to indicate the hostname to which it is trying to connect at the start of the TLS handshake. This is used in environments where multiple domains are hosted on the same IP address, allowing the server (or proxy) to present the correct certificate for the requested hostname. Thus, by combining the use of a proxy with proper SNI support, one can ensure both secure and accurate TLS termination. The proxy acts as a centralized control point : functioning not only as a basic firewall by filtering traffic (proxy main function), but also as a load balancer by routing requests to the appropriate backend, based on the requested hostname (SNI main function). 3.3.1 Proxy modes You can configure your proxy in two different ways: •TLS Passthrough: In this mode, the proxy does not decrypt the traffic. It simply forwards the encrypted TLS packets to the backend server. The TLS handshake and certificate exchange occur directly between the client and the backend server. Since the proxy does not see the decrypted content, it doesn’t need access to the TLS certificates. •TLS Termination: In this mode, the proxy decrypts the incoming TLS connection, acting as the endpoint of the TLS handshake. It requires a valid certificate for the requested domain, which it selects using the SNI provided by the client. After decryption, the proxy can inspect, log, or modify traffic, and may re-encrypt it to the backend server. This mode enables better routing and security policies, but also places the responsibility of certificate management on the proxy. The following diagram shows the difference between these two modes. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 21 CERN openlab Report Figure 11: TLS Passthrough VS TLS terminaison We chose to implement TLS termination at the proxy level primarily for security reasons. By terminating TLS on the proxy, we ensure that all incoming traffic is decrypted and inspected before reaching the back-end servers. It also prevents external clients from establishing direct connections to backend services, isolating the internal infrastructure. 3.3.2 Proxy configuration with SNI Here is the code of the proxy configuration used during our proof of concept, to serve 3DNs, all three hosted on the same Virtual Machine, as depicted on the previous diagram : We will explain straight after the logic of such a configuration. Stored in /usr/local/etc/haproxy/haproxy.conf: global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 22 https https http http decryption here CERN openlab Report group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog timeout connect 10s timeout client 1m timeout server 1m maxconn 3000 #--------------------------------------------------------- # FRONTEND : TLS termination (avec SNI, plusieurs certs) #--------------------------------------------------------- frontend https_front bind *:443 ssl crt /etc/ssl/haproxy/ # here are all the .pem mode http option forwardfor http-request set-header X-Forwarded-Proto https use_backend backend_321_http if {ssl_fc_sni -i db-lb-k8s-321.cern.ch } use_backend backend_322_http if {ssl_fc_sni -i db-lb-k8s-322.cern.ch } use_backend backend_326_http if {ssl_fc_sni -i db-lb-k8s-326.cern.ch } #--------------------------------------------------------- # BACKENDS HTTP #--------------------------------------------------------- backend backend_321_http mode http server db1 <IP>:8080 check backend backend_322_http mode http server db2 <IP>:8181 check backend backend_326_http mode http server db3 <IP>:8282 check #certif stored at /etc/ssl/acme/db-lb-k8s-321.cern.ch/fullchain.pem #/etc/ssl/acme/db-lb-k8s-321.cern.ch/privkey.pem Here are the two most important things about the code : •We connected three backend servers using SNI (Server Name Indication). HAProxy inspects the hostname requested in the TLS handshake and chooses the appropriate backend based on that value. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 23 CERN openlab Report •The certificates are read by the proxy from the following path: /etc/ssl/haproxy/. Each certificate must be in .pem format, containing both the private key and fullchain certificate (we will specify in next paragraph what it is). They must be placed here manually or automatically (e.g., by configuring Let’s Encrypt or another ACME client to store or symlink the certs there). The following diagram sum up the mechanics : Figure 12: HAproxy configuration mechanics 3.3.3 Backend servers In order to assign each DN a service that could listen on the port, we decided to setup three simple python server on port 80,81 and 82. Here is the code of one of the server : from http.server import HTTPServer, SimpleHTTPRequestHandler PORT = 8080 server_address =('', PORT) httpd =HTTPServer(server_address, SimpleHTTPRequestHandler) print("Port {}".format(PORT)) httpd.serve_forever() 3.3.4 Final test Let’s put in a nutshell all the previous subjects we tackled in the following experiment : 1. In the VM are the acme.sh and the three Python servers, each related to a DN ; 2. Launch acme.sh to generate a certificate for each one of the three DNs, acme.sh will provide three files per DN : •the private key of the DN, to decrypt the incoming data ; UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 24 CERN openlab Report 4.2 Kubernetes environment 4.2.1 Roadmap The goal is to study the certificate management for the following kubernetes cluster : Figure 16: Kubernetes cluster You can see that the decryption is made at the step of the ingress controller and not at the step of the Loadbalancer. It enables more control on the data and a better secret. In order to work in TLS terminaison that way, the goal is to create some kubernetes secrets, defined in each ingress, and which contains the fullchain certificate+decryption key. Here are some useful information for the setup : •Create ingress on node worker : UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 31 CERN openlab Report kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controllerv1.10.1/deploy/static/provider/cloud/deploy.yaml •Set up an external NGINX loadbalancer, which only forward all the encrypted incoming data to the ingress controller (we will provide the configuration later) •Make sure that you have stored the fullchain and the key of each DN somewhere in the cluster. You can then create secret associated with the certificates with the following commands : kubectl create secret tls <name-of-the-secret> –cert=<path-to-fullchain> –key=<path-to-key> -n default •Each ingress must be in the same namespace than the TLS secret !! •If one secret is changed, the ingress controller needs to be reloaded, and it can take time before everything works again : kubectl rollout restart deployment ingress-nginxcontroller -n ingress-nginx Official documentation : https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ 4.2.2 Configuration files Here are the configurations of several components of the cluster : Ingress Controller apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of"> creationTimestamp:"2025-08-07T07:22:43Z" labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.10.1 name: ingress-nginx-controller namespace: ingress-nginx resourceVersion:"508909" uid: c8f2731d-e2a1-48c1-a9ca-075eed981a12 spec: clusterIP: 10.254.52.82 clusterIPs: - 10.254.52.82 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 32 CERN openlab Report ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: -name: http nodePort: 30080 port: 80 protocol: TCP targetPort: http -name: https nodePort: 30443 port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx sessionAffinity: None type: NodePort status: loadBalancer: {} Ingress ressource, 1 conf file for all three ingresses # ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: sites-ingress namespace: default annotations: nginx.ingress.kubernetes.io/ssl-redirect:"true" spec: ingressClassName: nginx tls: -hosts: - db-lb-k8s-321.cern.ch secretName: tls-321 -hosts: - db-lb-k8s-322.cern.ch secretName: tls-322 -hosts: - db-lb-k8s-326.cern.ch secretName: tls-326 rules: UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 33 CERN openlab Report -host: db-lb-k8s-321.cern.ch http: paths: -path: / pathType: Prefix backend: service: name: db-lb-k8s-321-cern-ch port: number: 80 -host: db-lb-k8s-322.cern.ch http: paths: -path: / pathType: Prefix backend: service: name: db-lb-k8s-322-cern-ch port: number: 80 -host: db-lb-k8s-326.cern.ch http: paths: -path: / pathType: Prefix backend: service: name: db-lb-k8s-326-cern-ch port: number: 80 Pod configs apiVersion: v1 kind: Pod metadata: name: db-lb-k8s-321-cern-ch labels: app: db-lb-k8s-321-cern-ch spec: containers: -name: nginx image: nginx volumeMounts: -name: html mountPath: /usr/share/nginx/html volumes: -name: html configMap: UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 34 CERN openlab Report name: db-lb-k8s-321-cern-ch-html --- apiVersion: v1 kind: ConfigMap metadata: name: db-lb-k8s-321-cern-ch-html data: index.html: | <html><body><h1>Welcome on db-lb-k8s-321</h1></body></html> --- apiVersion: v1 kind: Service metadata: name: db-lb-k8s-321-cern-ch spec: selector: app: db-lb-k8s-321-cern-ch ports: -port: 80 targetPort: 80 LoadBalancer config files load_module /usr/lib64/nginx/modules/ngx_stream_module.so; events { worker_connections 1024; } stream { upstream ingress_https { server <IP>:30443; } server { listen 443; proxy_pass ingress_https; } } UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 35 CERN openlab Report 4.2.3 Results Figure 17: Secured connexion from the kubernetes cluster. Figure 18: GET https logs for 321 Figure 19: Kubernetes secrets 4.2.4 Debug commands I used •Check that the good certificate is used by the DN : curl -vk https://db-lb-k8s321.cern.ch ] •Check that secrets are read : kubectl logs ingress-nginx-controller-6bb485db4b-95dpb -n ingress-nginx | grep tlsOfficial documentation : https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 36 CERN openlab Report 5 Conclusion 5.1 Pros acme.sh Here is a list of advantages of using acme.sh script, which I could experienced during this project : •No dependencies : Unlike other ACME clients, acme.sh is written in shell script, meaning there is no need to install other heavy dependencies. This makes it particularly convenient for environments where lightweight tooling is preferred. •Flexible configuration options: acme.sh supports multiple certificate types, including wildcards and multi-domain certificates, and allows custom configurations through a simple script interface. We have seen it with the use of our custom API along this project. •Support for HTTP and TLS-ALPN Challenges: In addition to DNS challenges, acme.sh also supports HTTP-01 and TLS-ALPN-01 challenges, providing more flexibility for certificate validation depending on the environment or specific use case. We did not try to configure other type of challenge during this project, but one could find many online ressources explaining how to implement it. •Open-Source: acme.sh is an open-source project, meaning that users can contribute to its development or report bugs, and they benefit from regular updates and improvements from the community. 5.2 Limits of acme.sh We could not witness many bugs on the script, and this section could deserve an entire audit. But still, two issues caught our attention : 5.2.1 TXT record not empty on challenge subdomain The first point I struggled with is the following one : ACME protocol used with dns-01 is based on adding TXT records, as we have seen it. During test sessions, many TXT records were added and removed. What we observed is that if the TXT record field is not blank before launching the acme.sh script, the TXT record will never be registered in the DNS, and the command will end up in an endless loop stuck at the public DNS verification state. This issue is tricky because it was never explained in the logs of the command, and can be even more complicated to spot because of the DNS time propagation. Conclusion : Always launch acme.sh script with a wiped TXT record zone. 5.2.2 Openssl showcert with SNI disabled. The issue By using –noservername option with the openssl command, meaning asking to see a certificate while disabbling the SNI, we could still achieve to get one certificate, as shown in the following screenshot : UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 37 CERN openlab Report Figure 20: Showcert, SNI disabled. This issue could be observed for both proxy types (HAproxy and Apache). What is happening By deactivating the SNI, we are querying the proxy without mentioning which DN we want to query. In that context, the Proxy respond with a default certificate : the first one in alphabetical order OR first one in the config file (depending on the Linux UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 38 CERN openlab Report distribution you use). Why is it an issue The core issue stems from the reliance on Server Name Indication (SNI) to serve the correct SSL/TLS certificate in environments hosting multiple domains on the same IP address. When SNI is disabled or unsupported-as may be the case with some legacy systems, IoT devices, or outdated clients-the server defaults to returning a fallback certificate, which may not match the requested domain. This can result in trust errors, failed connections, or unintended exposure of the default certificate. Solution •Configure a default backend named 000-default.conf which will be served by default. The HTML file linked with this backend could provide an explanation page pleasing the client to use a device supporting SNI. •Use wildcard certificates : or at least configure your proxy to serve a wildcard certificate in case the client does not support SNI. This can be done by using a wildcard certificate on the first subdomain (the one that would be queried if SNI is down). NB : We also tried to display a 403 error by adding the following line to the Apache configuration file (apache2.conf, or httpd.conf): SSLStrictSNIVHostCheck on. But this can’t work as our proxy works in TLS terminaison : SNI required message defined in the fallback VirtualHost is part of the HTTP layer, but in this case the TLS handshake either fails or does not proceed beyond the initial negotiation due to the missing SNI. In our context — where Apache acts as a TLS termination proxy and forwards decrypted traffic to backends over plain HTTP — this means that the 403 error page can only be served after successful decryption of the TLS session. Since we are trying to return an HTTP error before TLS decryption is even possible, it will never be delivered to the client. Default backend configuration About the default configuration page : •it is using a self-signed certificate for our POC, but could use a real certificate. Ours was generated via the following command : openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/pki/tls/private/fallback.key -out /etc/pki/tls/certs/fallback.crt -subj "/CN=default-fallback". •it provides a fallback when the client does not send SNI, create a dedicated VirtualHost listening on port 443 with SSL enabled (SSLEngine on), •one can check that the expected error page is correctly served with : printf "GET / HTTP/1.1\r\nHost: db-lb-k8s-321.cern.ch\r\nConnection: close\r\n\r\n" | openssl s_client -connect db-lb-k8s-321.cern.ch:443 -noservername -quiet NB : The same issue can be solved with the same steps for an HAproxy, as mentioned here : https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension Finally, here is the html code of the default backend server : UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 39 CERN openlab Report frontend https-in bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1 mode http acl no_sni req_ssl_sni -m len 0 http-request deny deny_status 403 if no_sni default_backend my_backend 5.3 What is next ? 5.3.1 More automation As we said, there is still one manual step remaining: in this POC, we had to concatenate for each DN the private key + fullchain.pem and put it in the right file in haproxy. This step can be automated natively by Let’s Encrypt, using the – reloadcmd option. This option will launch every command you decide to write. For instance, we could imagine doing this : --reloadcmd "scp /etc/ssl/acme/<DN>/* user@haproxy-vm:/etc/haproxy/ \\ && systemctl reload haproxy" This stands as a simple example, and it takes for granted that no step of authentification is required between the VM (hosting certificates and acme script) and the proxy. 5.3.2 About Wildcard certificates Let’s shed a light on wildcard certificates, since they were part of our research. The goal is to understand what they are and why we did not use it. Definition A wildcard certificate is a type of public key certificate (like SSL/TLS) that secures a primary domain and all its first-level subdomains. Instead of needing a separate certificate for each subdomain, a single wildcard certificate can be used, saving time and cost, especially for businesses with multiple subdomains. The certificate is identified by an asterisk (*) in the domain name, such as *.example.com. Case of use At first sight, this kind of certificate seems really convenient. Moreover, it could solve our issue about CNAME (see 3.2). But in reality, this kind of certificate can raise many security issues such as : •Private key compromission : the private key associated with the wildcard certificate is shared across all secured subdomains. This means if the private key is compromised, all those subdomains are vulnerable. •Limited control over individual subdomains: since wildcard certificates apply to all subdomains under a primary domain, you lose the ability to manage each subdomain individually. For example, if one subdomain is compromised or needs a change in security settings (e.g., key rotation), you would have to update the certificate for all other subdomains too, which can be problematic. UNIFIED TLS CERTIFICATE AUTOMATION USING LET’S ENCRYPT 40