CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE

CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE September 2025 AUTHOR(S): Dawid Grabowski CERN, Storage and Data Management Group, IT Departament SUPERVISOR(S): Enrico Bocchi Abhishek Lekshmanan CERN openlab Report ABSTRACT The long-term storage and availability of vast datasets, such as those generated by the Large Hadron Collider (LHC), are critical to CERN’s scientific mission. The Ceph distributed storage system, with its RADOS Gateway (RGW) S3-compatible object storage interface, provides a scalable and resilient solution. To ensure high availability and disaster recovery, RGW can be deployed in a multisite replication configuration, for instance, between the Meyrin and Prévessin data centers. However, maintaining perfect data consistency across geographically distributed sites presents a significant challenge. Latency, network partitions, or software bugs can lead to replication inconsistencies, where data exists at one site but is missing or outdated at another. This project addresses this challenge through the development of the Ceph RGW Multisite Consistency Monitor, a comprehensive tool designed to detect and diagnose replication discrepancies. The tool operates in two distinct modes: a non-intrusive Passive Monitoring Mode that listens to real-time S3 operations via Ceph’s Kafka-based bucket notifications, and an Active Testing Mode that generates controlled S3 workload (PUT/DELETE operations) to stress-test the replication pipeline and validate consistency under load. The system leverages the AWS S3 command-line interface for object manipulation and a high-performance C++ component for real-time Kafka event processing. By comparing the ground truth of performed S3 operations with the stream of replication notifications, the monitor can pinpoint specific inconsistencies, such as missing notifications, extra notifications, and orphaned synchronization events. The tool produces detailed JSON reports and human-readable summaries, providing storage administrators with the necessary diagnostics to maintain the integrity of CERN’s distributed storage infrastructure. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 1 CERN openlab Report TABLE OF CONTENTS 1 INTRODUCTION 4 1.1 The Role of Distributed Storage at CERN . . . . . . . . . . . . . . . . . . . . . 4 1.2 Ceph and the RADOS Gateway (RGW) . . . . . . . . . . . . . . . . . . . . . . 4 1.3 The Challenge of Multisite Replication Consistency . . . . . . . . . . . . . . . . 4 1.4 ProjectObjectives .................................. 4 2 Project Specification 5 2.1 CoreRequirements .................................. 5 2.2 FunctionalSpecifications............................... 5 2.3 Non-Functional Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.4 ScopeandLimitations ................................ 6 3 Background and Core Technologies 6 3.1 Ceph RGW Multisite Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 The Role of Bucket Notifications with Kafka . . . . . . . . . . . . . . . . . . . . 6 3.3 Interacting with Ceph via the S3 API . . . . . . . . . . . . . . . . . . . . . . . . 7 4 System Architecture and Design 7 4.1 High-Level Architectural Overview . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2 ModularStructure .................................. 7 4.3 Configuration Management System . . . . . . . . . . . . . . . . . . . . . . . . . 8 4.4 C++ Event Watcher Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5 Implementation Details: Operational Modes 9 5.1 ActiveTestingMode ................................. 9 5.1.1 Workflow ................................... 9 5.1.2 Configuration................................. 9 5.1.3 UsageExample ................................ 10 5.2 PassiveMonitoringMode............................... 10 5.2.1 Workflow ................................... 10 5.2.2 Real-time Inconsistency Detection . . . . . . . . . . . . . . . . . . . . . 11 5.2.3 UsageExample ................................ 11 6 Inconsistency Detection Logic 11 6.1 TheAnalysisPipeline ................................ 11 6.2 Types of Inconsistencies Detected . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6.3 Data Structures and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 7 Ancillary Tooling: S3 Event Generator 12 7.1 PurposeandFeatures................................. 12 7.2 OperationalModes .................................. 13 7.3 Integration with Passive Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . 13 8 Output, Reporting, and Results 13 8.1 DirectoryStructure.................................. 13 8.2 ActiveModeReports................................. 13 8.3 PassiveModeReports ................................ 14 CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 2 CERN openlab Report 8.4 InterpretingResults.................................. 14 9 Conclusion and Future Work 14 9.1 Development Challenges and Lessons Learned . . . . . . . . . . . . . . . . . . . 14 9.1.1 S3 Notification Incompatibility . . . . . . . . . . . . . . . . . . . . . . . 14 9.1.2 Monolithic Code Growth . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 9.1.3 Regression and Lack of Automated Testing . . . . . . . . . . . . . . . . . 15 9.1.4 Python Concurrency and Performance Bottlenecks . . . . . . . . . . . . 15 9.1.5 Uncontrolled Scope Creep . . . . . . . . . . . . . . . . . . . . . . . . . . 15 9.2 Project Summary and Achievements . . . . . . . . . . . . . . . . . . . . . . . . 16 9.3 FutureEnhancements................................. 16 10 References 16 CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 3 CERN openlab Report 1 INTRODUCTION 1.1 The Role of Distributed Storage at CERN The European Organization for Nuclear Research (CERN) operates the world’s largest particle physics laboratory. The experiments conducted, particularly at the Large Hadron Collider (LHC), generate petabytes of data annually[4]. The reliable storage, rapid access, and longterm preservation of this data are paramount. Distributed storage systems are essential to meet these demands, offering scalability, fault tolerance, and high performance that monolithic systems cannot provide. 1.2 Ceph and the RADOS Gateway (RGW) Ceph is an open-source, software-defined storage platform that provides object, block, and file storage from a single unified cluster. Its foundation is the Reliable Autonomic Distributed Object Store (RADOS)[6]. For object storage, Ceph provides the RADOS Gateway (RGW), which presents a RESTful API compatible with Amazon S3 and OpenStack Swift. This allows applications and services at CERN to interact with Ceph using a widely adopted and standardized interface.[3][2] 1.3 The Challenge of Multisite Replication Consistency To ensure business continuity and for disaster recovery, critical data must be replicated across multiple physical locations. Ceph RGW supports a multisite configuration[3], where data written to a primary site is asynchronously replicated to one or more secondary sites. For CERN, this could involve replicating data between the primary data center in Meyrin, Switzerland, and a secondary site in Prévessin, France. While asynchronous replication enhances write performance, it introduces the risk of data inconsistency. Potential issues include: •Replication Lag: Events from the primary site are delayed in reaching the secondary site. •Lost Updates: A network partition or service failure could cause a notification to be lost entirely, resulting in an object not being replicated. •Orphaned Events: A replication event might be received at a secondary site for an operation that never completed or was never initiated at the primary site. These inconsistencies can be silent and difficult to detect, potentially leading to data loss or access to stale data. A robust monitoring tool is therefore not just beneficial but essential for operational confidence. 1.4 Project Objectives The primary objective of this project was to design, implement, and document a tool to monitor and validate the consistency of Ceph RGW multisite replication. The tool needed to be capable of both passively observing production traffic and actively generating synthetic workloads to probe the system for weaknesses. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 4 CERN openlab Report 2 Project Specification 2.1 Core Requirements The project was defined by the following core requirements: •Detect Replication Inconsistencies: The primary goal is to identify discrepancies between S3 operations performed on a Ceph cluster and the resulting replication state across sites. •Produce Actionable Reports: The output must be clear, detailed, and provide administrators with the information needed to diagnose and resolve any detected inconsistencies. •Provide Dual Operational Modes: The tool must support two fundamental modes of operation: – Active Testing: To generate a controlled S3 workload and verify that all operations are correctly replicated. – Passive Monitoring: To listen to S3 event notifications from an existing, live environment without generating its own traffic. 2.2 Functional Specifications •Active Mode: –Generate a configurable number of S3 objects. –Perform concurrent PUT operations to upload these objects. –Wait for a configurable period to allow for replication (object_lifetime_seconds). –Perform concurrent DELETE operations to clean up the objects. –Capture all corresponding Kafka notifications. –Compare the set of S3 operations with the set of Kafka notifications, additionally, compare master site and secondary site notifications to generate a consistency report. •Passive Mode: –Connect to a specified Kafka topic. –Continuously consume and parse S3 event notifications. –Provide a real-time console display of event statistics (e.g., event rate, object count). –Detect and log inconsistencies in real-time, such as orphaned synchronization events. –Run indefinitely until manually stopped. •Configuration: All parameters (S3 endpoints, credentials, Kafka servers, object counts, etc.) must be configurable via a JSON file. –Command-line arguments must override settings from the configuration file. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 5 CERN openlab Report 2.3 Non-Functional Specifications •Performance: The passive monitoring mode must be lightweight and efficient, capable of handling a high rate of events without impacting the monitored systems. A highperformance C++ component should be used for the most performance-critical task of event parsing. •Utilize Ceph Bucket Notifications: The detection mechanism must be built upon the native Ceph RGW feature that publishes object operation events to a Kafka topic. •Usability: The tool should be easy to set up and run, with clear command-line options and comprehensive documentation. •Modularity: The codebase must be well-structured, separating concerns such as S3 management, Kafka interaction, and consistency analysis into distinct modules. 2.4 Scope and Limitations •The tool focuses on detecting inconsistencies at the notification level. It confirms that create and delete notifications are successfully propagated for each S3 operation. It does not perform byte-by-byte data integrity checks on the object content itself. •The tool relies on the aws-cli being correctly installed and configured for S3 operations. •Automatic remediation of inconsistencies is outside the scope of this project. The tool’s role is detection and reporting. 3 Background and Core Technologies 3.1 Ceph RGW Multisite Architecture A Ceph multisite configuration consists of a master zonegroup and one or more secondary zonegroups. Every zonegroup comprises one or more zones. Data replication occurs within zonegroups, while metadata is transferred from the master zone of a zonegroup to the other zonegroups. This replication is asynchronous and relies on RadosGW daemon processes running at each site to read the logs and apply the changes.[3] 3.2 The Role of Bucket Notifications with Kafka A significant aspect of contemporary Ceph RGW is its capability to publish notifications regarding bucket and object events to external endpoints, which can include HTTP, AMQP, and Kafka. This project takes advantage of the integration with Apache Kafka. When an object in a certain bucket is created, deleted, or modified, RGW publish a formatted notification to a designated Kafka topic. This mechanism is central to the monitor’s design. The stream of Kafka messages provides an authoritative log of operations as seen by Ceph itself. The event messages distinguish between primary operations (e.g., ObjectCreated:Put) and replication events (ObjectSynced:Create), which is crucial for detailed analysis. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 6 CERN openlab Report 3.3 Interacting with Ceph via the S3 API The RADOS Gateway provides an S3-compatible API, which has become the de facto standard for object storage.[1] This allows the use of a vast ecosystem of existing tools. This project uses the official AWS Command Line Interface (aws-cli) to perform S3 operations.[5] As seen in src/core/s3_manager.py, operations are executed via asyncio subprocess calls to aws s3 cp and aws s3 rm, which provides a robust and well-tested method for interacting with the cluster. 4 System Architecture and Design 4.1 High-Level Architectural Overview The monitor is designed with a modular, service-oriented architecture. A central entry point (src/monitor.py) parses user input and delegates control to one of two main service classes: ActiveTestingService or PassiveListeningService. These services utilize a shared set of core modules for handling configuration, S3 operations, Kafka communication, and reporting. Entry Point (monitor.py) Services Layer ActiveTestingService - Generates S3 load - Analyzes results PassiveListeningService - Listens to Kafka - Real-time monitor Core Modules ConfigManager S3Manager KafkaManager ResultsManager ConsistencyAnalyzer Figure 1: System Architecture Diagram 4.2 Modular Structure •src/monitor.py: The main CLI entry point. Handles argument parsing and service instantiation. •src/services/: Contains the primary business logic for each operational mode. –active_testing_service.py: Orchestrates the entire active test lifecycle. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 7 CERN openlab Report –passive_listening_service.py: Manages the continuous Kafka monitoring loop and C++ watcher integration. •src/core/: Provides shared, reusable functionality. –config.py: Manages loading and validating configuration. –s3_manager.py: An abstraction layer for performing S3 operations using aws-cli. –kafka_manager.py: Handles starting/stopping local Kafka services and consuming messages from topics. –consistency_analyzer.py: The core engine that compares S3 and Kafka data to find inconsistencies. –reporting.py: Manages the creation of output directories and report files. 4.3 Configuration Management System To provide flexibility, the monitor uses a three-tier priority system for configuration, ensuring that users can easily override defaults for specific runs: 1. Built-in Defaults (Lowest Priority): Sensible default values are hard-coded. 2. JSON Configuration File (Medium Priority): A user-provided file (--config) specifies the bulk of the environment settings. 3. CLI Arguments (Highest Priority): Any parameter provided on the command line (e.g., --s3-objects 50) overrides values from the config file and defaults. This hierarchy is managed by the ConfigManager class in src/core/config.py. 4.4 C++ Event Watcher Integration Python, while excellent for orchestration, can be a bottleneck for high-throughput, low-latency tasks like parsing millions of log lines. In passive mode, where the monitor might process thousands of events per second, performance is critical. To address this, the PassiveListeningService integrates a dedicated C++ component. This component is automatically compiled and launched by the monitor. Its responsibilities are: •Connecting directly to the Kafka log file being written by the consumer. •Parsing event lines in real-time with high efficiency. •Performing immediate inconsistency detection (e.g., finding an ObjectSynced event without a preceding primary event). •Writing structured output to realtime_events.jsonl and inconsistencies.jsonl with a configurable flush interval (watcher_flush_interval_ms). If the C++ component fails to compile or run, the system gracefully falls back to a pure Python implementation, ensuring functionality at the cost of performance. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 8 CERN openlab Report •Lesson Learned: Establishing clear component boundaries from the project’s outset is critical for scalability. Adopting a component-based architecture, even for initial prototypes, mitigates significant refactoring efforts and reduces errors during development. A more planned approach to architectural design can prove highly beneficial. 9.1.3 Regression and Lack of Automated Testing •Problem: During rapid development, new feature additions frequently broke existing functionality. These regressions were time-consuming to trace manually and slowed down progress. •Solution: A comprehensive testing framework was implemented, featuring unit tests for core logic, component tests, and end-to-end (E2E) tests that validate the entire workflow with live Kafka streams. The full test suite was integrated into a GitLab CI/CD pipeline. •Lesson Learned: A robust CI/CD pipeline is essential for maintaining code stability. Adopting a Test-Driven Development (TDD) approach, where tests are written before or alongside new features, is the most effective strategy for preventing regressions and ensuring software quality. 9.1.4 Python Concurrency and Performance Bottlenecks •Problem: The pure Python implementation struggled to process high-frequency event streams in the passive monitoring mode. Due to performance limitations and the Global Interpreter Lock (GIL), events were sometimes processed out of order or missed entirely during high-load scenarios. •Solution: The performance-critical, real-time event parsing logic was migrated from Python to a dedicated, high-performance C++ component (kafka_event_watcher). •Lesson Learned: For applications where performance, low latency, and precise timing are crucial, Python may not be the optimal tool for all tasks. C++ is better suited for high-throughput processing, and a hybrid architecture—using Python for high-level orchestration and C++ for critical components—can provide the best of both worlds. 9.1.5 Uncontrolled Scope Creep •Problem: The project’s scope expanded significantly beyond the original requirements. What was initially conceived as a simple monitoring tool evolved into a complex platform. – Original Goal: A service to read a Kafka stream, parse events, analyze inconsistencies, and log the status. –Final Implementation: A full platform that could auto-install and configure Kafka, manage test infrastructure, perform active stress tests (upload, sync wait, delete), and build its own components, in addition to the original passive monitoring goal. •Lesson Learned: Defining and adhering to clear scope boundaries is paramount for successful project management. Using a prioritization framework like MoSCoW (Must have, Should have, Could have, Won’t have) helps resist the urge to "just add one more feature" and ensures that core objectives are met on time and within the specified constraints. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 15 CERN openlab Report 9.2 Project Summary and Achievements This project successfully delivered the Ceph RGW Multisite Consistency Monitor, a tool that meets all the core requirements outlined in the project specification. It provides robust, flexible, and performant capabilities for detecting replication inconsistencies in a Ceph multisite environment. The dual-mode approach allows it to serve both as a forensic tool for production environments and as a validation tool for staging and development. The modular architecture and documentation ensure its maintainability and extensibility. The tool represents a significant step forward in ensuring the operational reliability of CERN’s distributed object storage infrastructure. And it will certainly serve as a proof of concept for collecting data from cluster operations using the Ceph feature of bucket notifications 9.3 Future Enhancements While the current tool is fully functional, several enhancements could further increase its value: •Multiple buckets notification collection: The system should be adapted to support multiple buckets, currently it is adjusted to one, additionally it must be named the same on two clusters, which is a simplification used in the project, but is easy to expand. •Object-lifetime based inconsistency marking: In passive mode, the “inconsistency” flag should occur after time n. In program, an object is inconsistent immediately after being added because it hasn’t synced yet, which is true. However, a synchronization could occur within 10 seconds, and if this information would be sent as an alert; after a short while, it becomes outdated. The time for defining something as inconsistent should be, for example, 20 times the synchronization time in the configuration, e.g., 100 seconds. •Web-based UI: A graphical dashboard could provide a more intuitive way to view realtime statistics and browse inconsistency reports. •Prometheus Integration: Exposing key metrics (e.g., inconsistency counts, event rate) in a Prometheus-compatible format would allow for integration into CERN’s standard monitoring and alerting systems. •Automated Remediation Scripts: The tool could be extended to generate suggested remediation commands (e.g., commands to manually sync or delete an object) based on the inconsistencies it finds. •Broader Protocol Support: While focused on Kafka, the architecture could be adapted to support other notification endpoints like AMQP or HTTP. 10 References [1] Amazon Web Services, Inc. AWS CLI Command Reference - s3. 2025. url:https:// awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html (visited on 09/04/2025). [2] Ceph Development Team. Ceph RADOS Gateway Bucket Notifications. 2025. url:https: //docs.ceph.com/en/latest/radosgw/bucket-notifications/ (visited on 09/04/2025). [3] Ceph Development Team. Ceph RADOS Gateway Multisite Replication. 2025. url:https: //docs.ceph.com/en/latest/radosgw/multisite/ (visited on 09/04/2025). CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 16 CERN openlab Report [4] CERN. Cern Copmuting.url:https://www.home.cern/science/computing. [5] Dawid Grabowski. Ceph S3 Bucket Notifications Service. CERN, Sept. 2025. url:https: //gitlab.cern.ch/ceph/s3-notifications (visited on 09/05/2025). [6] Sage A. Weil et al. “RADOS: A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters”. In: Proceedings of the 2nd International Workshop on Petascale Data Storage (PDSW ’07). New York, NY, USA: ACM, 2007. doi:10.1145/1374596.1374606. CEPH RGW MULTISITE CONSISTENCY MONITORING SERVICE 17