Exploring Oracle Cloud for Disaster Recovery

AUGUST 2024 Exploring Oracle Cloud for Disaster Recovery AUTHOR: Victor Virag IT-PW SUPERVISOR: Hubert Odziemczyk Exploring Oracle Cloud for Disaster Recovery 2 CERN openlab Report // 2024 PROJECT SPECIFICATION CERN IT Disaster Recovery Improvement Project Current System: Platform: JEEDY (Java/Kubernetes platform for web applications) Data: EDMS Storage: NFS with snapshots, tape backups Backup Method: OpenStack Object Storage Project Aim: Backup Replication and Immutability: Research replication of backups from OpenStack to Oracle Cloud Archive-tier Object Storage and protect them using retention policies. Implementation Strategy: Tools: Use restic to perform backups to OpenStack Object storage. Use Rclone for replicating backups from OpenStack to Oracle Cloud Archive-tier Object Storage. Benefits: Immutability and location separability of backups Faster individual file retrieval compared to tape storage Complementary secondary copy to existing tape backups Security and Monitoring: Retention Rules: Prevent unauthorized deletion or modification. Monitoring: Prometheus to provide real-time performance data. Goals: Ensure robust recovery in case of primary backup failure Improve data protection against malicious access Strengthen overall backup resilience and reliability  Exploring Oracle Cloud for Disaster Recovery 3 CERN openlab Report // 2024 ABSTRACT This project implements a backup replication and disaster recovery system for CERN’s JEEDY platform applications, including EDMS. Currently, backups are stored in OpenStack Object Storage, in addition to NFS snapshots and tape backups, which were the original methods. The project uses Rclone to copy restic backups from OpenStack to Oracle Cloud Archive-tier Object Storage. This method ensures backups are immutable, geographically seperate, and offers a secondary copy that is faster to retrieve compared to tape storage. Retention rules are applied to prevent unauthorized deletion or modification. Prometheus is used for real-time monitoring of system performance. This solution improves data protection, guards against malicious access, and ensures recovery if primary backups fail, improving the overall resilience and reliability of CERN’s backup strategy.  Exploring Oracle Cloud for Disaster Recovery 4 CERN openlab Report // 2024 TABLE OF CONTENTS INTRODUCTION 05 RESTIC BACKUPS 05 OPENSTACK INTEGRATION 06 ORACLE CLOUD REPLICATION 06 DISASTER RECOVERY PROCESS 07 PERFORMANCE MONITORING 07 CONCLUSION 09 ACKNOWLEDGEMENTS 09  Exploring Oracle Cloud for Disaster Recovery 5 CERN openlab Report // 2024 1. INTRODUCTION The disaster recovery system for the JEEDY platform, which originally included backups from services on NFS to tape storage, now also performs backups to CERN S3 storage hosted on OpenStack. This data is replicated across the Meyrin and Prévessin data centers, while the NFS storage continues to be backed up to tape. While this solution currently performs backups for some services, it does not yet cover the full range of services hosted on the Java Tomcat platform. The primary objective of this project is to extend the backup pipeline to replicate these backups from CERN S3 to Oracle S3 storage and evaluate how well these two storage solutions can work together. The work focuses on ensuring the immutability of backup objects stored on Oracle Cloud. This is necessary for ensuring backup integrity, and is achieved through the use of retention policies to prevent object modification. Additionally, monitoring components are used to ensure the backup pipeline operates smoothly and that any issues are identified and addressed promptly. To handle this, a Python script is used to collect metrics from the relevant Oracle APIs. These metrics are then formatted for Prometheus, which processes and displays them for ongoing monitoring and alerting. We also backfill Prometheus with historical data from restic logs to evaluate performance of real-world backups. Finally, Grafana is used to fetch metrics from Prometheus and display them in a well-formatted dashboard. 2. RESTIC BACKUPS Restic is the tool chosen for handling backups due to its robust features and security guarantees. It performs append-only backups, where data is encrypted using the AES-256-CTR symmetric encryption scheme, insuring data confidentiality. Restic also uses the Poly1305-AES Message Authentication Code (MAC) to also ensure data integrity. A key feature of Restic is its use of Content Defined Chunking (CDC). Unlike traditional fixed-size chunking, CDC allows Restic to create variable-sized blobs based on the actual content, optimizing storage efficiency. It uses a rolling hash to determine the starting offset of each blob, which is particularly useful when files undergo small modifications. The CDC approach minimizes the number of new blobs that need to be created, as existing blobs can often be reused by adjusting their offsets. The blobs are organized into larger pack structures, and index files are maintained to track them. Each backup operation generates a snapshot representing the state of the backup at a specific point in time. These snapshots are then used to restore the system to a particular configuration when needed. Over time, as files are modified or deleted, some blobs and packs become obsolete. These stale blobs can be cleaned up or "pruned" by deleting old snapshots, which reduces the total backup size. However, it's important to note that the pruning process cannot be performed if the objects are under a retention rules, which makes them immutable. In such cases, if versioning is enabled, a new version of the object would be created instead of modifying the existing one. We monitor restic job status and performance by parsing the output logs and extracting metrics like backup respoitory size and time taken for the backup job. The metrics are then sent to the Push Gateway service, which is then scraped by Prometheus.  Exploring Oracle Cloud for Disaster Recovery 6 CERN openlab Report // 2024 Figures 1 and 2 show historical data of the size and backup time of restic backups for the EDMS service. On a larger timescale patterns such as lower modification counts and backup times during weekends and holidays can be identified, which coincide with real-world expectations. 3. OPENSTACK INTEGRATION In OpenStack S3, retention rules are configured with object versioning for buckets, offering a flexible and secure backup strategy. Versioning ensures that as long as retention rules apply, older versions of objects are preserved, allowing for modifications or deletions to be handled safely by creating new versions. However, one limitation of OpenStack's implementation is that administrators retain the ability to delete entire buckets. This could potentially expose the system to risks if high-privilege accounts are compromised. Another limitation of OpenStack is that it supports only a single storage tier. This reduces flexibility in cost management, particularly when handling large amounts of infrequently accessed backup data, as it cannot be automatically moved to a more cost-effective tier like archival storage. 4. ORACLE CLOUD REPLICATION To address some of the limitations of OpenStack, the project replicates the CERN S3 buckets to Oracle S3 using rclone, a command-line program that facilitates data transfer between different cloud storage services. a. ARCHIVE STORAGE Oracle Cloud offers an Archive storage tier, which is significantly more cost-effective for storing backup data that is rarely accessed. However, this tier comes with the trade-off of a 1-hour delay when restoring a file before it can be accessed. This delay is manageable for disaster recovery scenarios where immediate access to backups isn't required. Restic, the backup tool used in this project, cannot interact directly with Archive storage because it requires read access to certain files to manage its repository. However, the replication process from CERN S3 to Oracle S3 using rclone is unaffected by this limitation. Since rclone does not need to read the contents of the files during replication, it seamlessly works with Archive storage. This makes Archive storage a suitable solution for maintaining cost-effective, off-site backups.  Figure 1: Restic process historical data Figure 2: EDMS historical data Exploring Oracle Cloud for Disaster Recovery 7 CERN openlab Report // 2024 b. RETENTION RULES Unlike OpenStack, Oracle Cloud does not support object versioning when retention rules are enabled. This means that once a retention rule is applied, the objects cannot be modified or deleted until the retention period expires. While this may seem restrictive, it actually enhances security by preventing any accidental or malicious changes to the backup data during the retention period. Furthermore, Oracle Cloud allows these retention rules to be locked, ensuring that even administrators cannot override them. This feature provides an additional layer of protection for off-site backups, making it an excellent option for ensuring the long-term integrity of disaster recovery data. Figure 3 depicts a high-level overview of the backup pipeline, with the new addition of Oracle Cloud replication being on the left with yellow background. 5. DISASTER RECOVERY PROCESS The disaster recovery process would be to first attempt to recover from OpenStack S3 backups, and if not possible it can attempt to recover from the replicated backups in Oracle S3. Given the nature of the Archive storage tier, a delay of 1h is expected in accessing the data, as well as the data transfer overhead from Oracle to the CERN sites. Restoration from these backups involves restoring the data from the Archive storage, then using Restic to recover individual files or all files. Recovering individual files from Oracle S3 is preferable to tape storage as non-sequential file reads are likely to be quicker even with the 1 hour retrieval delay. 6. PERFORMANCE MONITORING An important aspect of the project is to provide real-time monitoring capabilities for the backup pipeline, along with a summary of performance and storage usage over time. This data can be used to identify anomalies in the backup process, estiamate future storage requirements and provide indications of failures.  Figure 3: backup pipeline overview Exploring Oracle Cloud for Disaster Recovery 8 CERN openlab Report // 2024 a. EXPORTING ORACLE CLOUD METRICS Oracle Cloud does not natively expose metrics in a format that is directly compatible with Prometheus. To address this, a custom Python exporter script is used. This script interacts with Oracle's APIs to retrieve the desired metrics, such as storage usage, object count, and request latency. These metrics are then converted into a format that Prometheus can ingest. The exporter hosts an HTTP endpoint, from which Prometheus scrapes the metrics at regular intervals. This setup allows for real-time monitoring of the backup pipeline, ensuring that any anomalies or issues are quickly identified and addressed. We can see the extracted metrics of bytes usage and object count in the OCI buckets in figures 4 and 5 respectively. b. PUSH GATEWAY We use push gateway to accept metrics from ephemeral jobs such as restic statistics outputs and then host an HTTP endpoint for Prometheus to scrape. The Oracle Cloud exporter script can also be run as a cronjob to push metrics to push gateway, avoiding the hosting of two seperate HTTP endpoints. c. BACKFILLING HISTORICAL RESTIC DATA To examine backup performance patterns from existing data we use a script to extract metrics from historical restic logs over the past 8 months and transform into an OpenMetrics file. This historical data is then directly backfilled into the local Prometheus database using Promtool, giving us insights into past backup trends and anomalies. A limitation of this approach is the necessity of direct write access to the data directory which stores the local DB of the target Prometheus instance.  Figure 4: OCI bytes usage Figure 5: OCI object count usage Exploring Oracle Cloud for Disaster Recovery 9 CERN openlab Report // 2024 d. GRAFANA INTEGRATION Figure 6: Grafana Dashboard 7. CONCLUSION Finally, we setup Grafana to authenticate with Prometheus and import existing metrics for restic jobs and Oracle Cloud performance. This allows us to setup dashboards to conveniently monitor backup pipeline performance at a glance from a central Grafana instance. An example Grafana dashboard is shown in Figure 6. This project explores the prospect of extending JEEDY platform's disaster recovery capabilities by replicating backups to Oracle S3 storage. We develop several automation and monitoring tools, alongside a comprehensive documentation detailing interconnectivity with Oracle S3. Future work may focus on further automating the backup and monitoring processes, as well as extending the backup coverage to all services on the Java Tomcat platform. 8. ACKNOWLEDGEMENTS I am sincerely grateful to my supervisor, Hubert Odziemczyk, for his invaluable guidance and support throughout this project. I would also like to express my appreciation for the collaborative and supportive environment created by my teammates, Artur, Daniel, Adrian, Antonio, Tomasz and Mouad. Additionally, I am thankful for the enriching experiences and valuable insights gained from the OpenLab summer school program, made possible by the dedication and contributions of the staff and fellow students. 