POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER

POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER August 2025 AUTHOR(S): Sarthak IT-SD-TAB SUPERVISOR(S): Pablo Oliver Cortes CERN openlab Report 1/2025 PROJECT SPECIFICATION This project implements an S3 Glacier–compatible interface for the CERN Tape Archive (CTA), using NooBaa as the S3 frontend and the EOS Tape REST API as the backend connector. The integration bridges the gap between modern cloud storage tools and CERN’s large-scale tape infrastructure, enabling researchers to interact with tape storage using familiar S3 commands. The system introduces executables like migrate,recall,helper_migrate, and setup_noobaa that extend NooBaa’s NSFS Glacier functionality and ensure proper interaction with CTA. These components handle file archival, retrieval, validation, and monitoring. The project was developed and tested in a Kubernetes-based environment, with automated deployment tools, CI/CD integration, and bulk testing for thousands of files. Authentication and security were implemented using EOS tokens, ensuring safe multi-user access. POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 1 CERN openlab Report 1/2025 ABSTRACT The CERN Tape Archive (CTA) is the backbone of CERN’s long-term data storage, safeguarding hundreds of petabytes of scientific data on tape. While CTA is highly reliable, it lacks compatibility with modern cloud storage APIs, which limits its usability in today’s research workflows. This project addresses that gap by building an S3 Glacier–compatible interface to CTA using NooBaa. Through this integration, users can archive and retrieve data from tape using familiar S3 commands, without needing to understand the complexities of tape storage. The system introduces a set of custom executables that connect NooBaa with CTA via the EOS Tape REST API, ensuring that files are properly validated once written to tape and can be restored reliably when needed. The solution not only enables seamless S3-based access to tape storage but also helps unify CERN’s backup and archival systems under CTA, improving efficiency and reducing dependence on proprietary technologies. By combining the robustness of tape with the flexibility of cloudnative APIs, this work lays the foundation for future, more user-friendly storage solutions at CERN. POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 2 CERN openlab Report 1/2025 TABLE OF CONTENTS 1 INTRODUCTION 4 1.1 Objectives....................................... 5 2 SYSTEM ARCHITECTURE 5 2.1 ArchitecturalOverview................................ 5 2.2 SystemComponents ................................. 6 2.2.1 NooBaaS3Interface ............................. 6 2.2.2 Custom NSFS Glacier Executables . . . . . . . . . . . . . . . . . . . . . 6 2.2.3 Supporting Scripts and Infrastructure . . . . . . . . . . . . . . . . . . . . 6 2.2.4 EOS HTTP TAPE REST API Interface . . . . . . . . . . . . . . . . . . 7 3 IMPLEMENTATION 7 3.1 Development Environment Configuration . . . . . . . . . . . . . . . . . . . . . . 7 3.2 NooBaa Deployment and Configuration . . . . . . . . . . . . . . . . . . . . . . . 7 3.2.1 InstallationProcess.............................. 7 3.2.2 The Configuration Challenge . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Authentication .................................... 8 3.3.1 TheSecurityChallenge............................ 8 3.3.2 The Two-Key Token System . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.4 CustomExecutables ................................. 9 3.4.1 The Migrate Executable: Archival with Validation . . . . . . . . . . . . . 9 3.4.2 The Recall Executable: Tape-to-Disk Staging Operations . . . . . . . . . 10 4 TESTING AND VALIDATION 13 4.1 TestStrategy ..................................... 13 4.2 BulkPerformanceTesting .............................. 14 4.3 CIPipelineIntegration................................ 14 5 RESULTS AND ACHIEVEMENTS 14 5.1 Successful Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.1.1 Complete S3 Glacier API Compatibility . . . . . . . . . . . . . . . . . . 15 5.1.2 Robust Archival Validation . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.1.3 Token-Based Authentication System . . . . . . . . . . . . . . . . . . . . 15 5.2 PerformanceMetrics ................................. 15 6 CHALLENGES AND SOLUTIONS 15 6.1 Technical Implementation Challenges . . . . . . . . . . . . . . . . . . . . . . . . 15 6.1.1 NooBaa NSFS Glacier Limitations . . . . . . . . . . . . . . . . . . . . . 15 6.1.2 Container Registry Rate Limiting . . . . . . . . . . . . . . . . . . . . . . 16 6.1.3 Archival Validation Complexity . . . . . . . . . . . . . . . . . . . . . . . 16 6.2 Authentication Architecture Challenges . . . . . . . . . . . . . . . . . . . . . . . 16 6.2.1 Multi-Tier Permission Requirements . . . . . . . . . . . . . . . . . . . . 16 6.2.2 Token Lifecycle Management . . . . . . . . . . . . . . . . . . . . . . . . . 16 7 CONCLUSION 16 8 References 17 POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 3 CERN openlab Report 1/2025 1 INTRODUCTION The CERN Tape Archive (CTA) is the backbone of CERN’s long-term scientific data preservation strategy. It manages hundreds of petabytes of data stored across thousands of tape cartridges, ensuring that physics data collected from experiments such as those at the Large Hadron Collider (LHC) remain accessible for decades. While tape storage offers durability and cost-effectiveness for archival, it does not natively support modern cloud storage interfaces such as the Amazon S3 API, which are increasingly required by researchers and applications. To address this limitation, this project integrates NooBaa, an object storage platform that provides an S3-compatible interface, with CTA through the EOS Tape REST API. This integration creates an S3 Glacier–compatible interface for CERN’s tape infrastructure, allowing researchers to archive and retrieve data using the same tools they would use with public cloud providers, while still leveraging CERN’s robust tape systems. At the core of this project is the development of custom executables that extend NooBaa’s incomplete Glacier functionality. These executables implement key operations such as archival, retrieval, monitoring, and lifecycle management, bridging the gap between NooBaa’s object storage model and CTA’s file-based tape backend. Beyond functionality, the project also emphasizes security, scalability, and usability. Authentication is handled via EOS tokens, ensuring that access controls remain consistent with CERN’s security policies. The system was deployed and tested in a Kubernetes environment, with CI/CD pipelines to validate workflows, bulk testing to handle thousands of files, and automated deployment tools to streamline setup across multiple servers. By combining cloud-native interfaces with tape-based storage, this work provides a foundation for future research workflows at CERN, enabling scientists to interact with archival data using familiar S3 APIs while ensuring reliability, efficiency, and long-term preservation. Figure 1 POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 4 CERN openlab Report 1/2025 1.1 Objectives •Fill the Missing Pieces: Complete the unfinished NooBaa NSFS Glacier functionality with production-quality code that could handle real-world workloads •Speak Everyone’s Language: Create seamless S3 Glacier API compatibility so applications could archive and retrieve data •Trust but Verify: Build robust validation systems to ensure files actually make it to tape, not just get uploaded to a staging area •Scale with Confidence: Support bulk operations handling thousands of files simultaneously •Integrate Seamlessly: Plug into CTA’s existing CI pipeline to ensure our code stays reliable as the system evolves 2 SYSTEM ARCHITECTURE 2.1 Architectural Overview The architecture consists of several key components. The S3 client serves as the interface through which users or systems send data storage and retrieval requests. NooBaa middleware acts as the S3-compatible gateway, receiving API requests, orchestrating the data flow, and managing the interaction between the client and the CERN storage backends. EOS acts as a high-speed SSD-based buffer, temporarily holding data before it is moved to or retrieved from the long-term tape storage managed by CTA. The data flow is separated into archival and retrieval workflows. When archival data is uploaded, the S3 client makes a request using the Glacier storage class, which NooBaa processes in two phases: an initial upload to EOS buffer and subsequent archival to CTA through migration scripts. For retrieval, the S3 client requests restoration, triggering NooBaa to invoke recall scripts that stage data from CTA tape to the EOS buffer before the data is transmitted back to the client. This process is secured through EOS tokens, which NooBaa uses to authenticate and authorize requests to EOS and CTA. This unified approach offers several advantages. By combining backup and archival storage into a single tape infrastructure accessible via a standard S3 Glacier interface, CERN achieves more efficient hardware usage and operational simplification. The use of open-source tools like NooBaa removes licensing constraints and allows full control of the software stack. The EOS buffer helps mitigate latency and performance challenges associated with tape storage. Ultimately, this solution provides CERN with a scalable, cost-effective, and flexible cold storage platform that can evolve with ongoing scientific data preservation requirements. Despite its benefits, there are some limitations to consider. NooBaa can exhibit unpredictable behavior during deployment, sometimes requiring troubleshooting. Additionally, certain processes such as the NSFS migration script require manual or scheduled execution, adding operational complexity. Furthermore, introducing an intermediate buffer layer and additional namespaces increases architectural complexity. Nevertheless, the successful testing of endto-end archival and retrieval operations demonstrates the viability and effectiveness of this POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 5 CERN openlab Report 1/2025 integrated S3 Glacier storage solution at CERN. The data flow follows a structured path through distinct system layers: S3 Client →NooBaa (S3 API) →Custom Executables →EOS HTTP TAPE REST API →CTA This layered approach ensures separation of concerns, with each component responsible for specific aspects of the integration while maintaining well-defined interfaces between system boundaries. 2.2 System Components 2.2.1 NooBaa S3 Interface NooBaa serves as the S3-compatible API gateway, providing standard Amazon S3 interface compatibility for client applications. The component handles comprehensive S3 operations including: •Bucket lifecycle management and policy enforcement •Object storage operations with GLACIER storage class recognition •RestoreObject API implementation for tape retrieval workflows •S3 lifecycle policy processing and automated storage class transitions 2.2.2 Custom NSFS Glacier Executables The integration implements two specialized executables that provide the missing NSFS Glacier functionality required for tape operations: 1. migrate: Handles file archival to tape storage with comprehensive validation through EOS archiveinfo API polling 2. recall: Manages file retrieval from tape storage through EOS staging operations and status monitoring 2.2.3 Supporting Scripts and Infrastructure The implementation includes several supporting scripts that enable system deployment, configuration, and operation: •generate_token: Manages dynamic generation and caching of EOS authentication tokens for secure API access •helper_migrate: Wrapper script that invokes the migrate executable with proper environment configuration and error handling •setup_noobaa: Comprehensive configuration script that deploys custom executables, patches StatefulSets, and configures NSFS Glacier parameters •install_noobaa: Automated deployment script that installs NooBaa using CERN container registry images to avoid Docker Hub rate limiting POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 6 CERN openlab Report 1/2025 2.2.4 EOS HTTP TAPE REST API Interface EOS provides the HTTP-based interface to CTA operations through implementation of the WLCG Tape REST API v1 specification. The interface supports: •File archival operations through HTTP PUT requests with authentication •Stage request submission and management for tape-to-disk operations •Archive information queries for file location and status verification •Operation status monitoring and progress tracking for long-running tape operations 3 IMPLEMENTATION 3.1 Development Environment Configuration The implementation employs a containerized, Kubernetes-based development environment designed to provide robust isolation between system components while supporting comprehensive integration testing. This architecture uses namespace-based separation to clearly delineate development activities from production operations, thereby safeguarding the integrity and stability of the CERN Tape Archive (CTA) system. The deployment is organized into three primary components: •CTA Environment (dev namespace): This namespace contains the core CTA components, including frontend services, tape daemon processes, and the scheduler infrastructure. It effectively mirrors the production environment and is shielded from development activities to ensure continuity and reliability of ongoing operations. •NooBaa Integration (noobaa namespace): Dedicated to the deployment of NooBaa along with its custom executables and configuration files, this namespace serves as the experimental playground for integration development and testing, isolated from the production workflow. •Container Registry (CERN Infrastructure): Leveraging CERN’s internal container registry mitigates external dependency constraints, such as Docker Hub’s rate limiting, thereby enhancing the efficiency and robustness of continuous integration pipelines. 3.2 NooBaa Deployment and Configuration 3.2.1 Installation Process The NooBaa deployment process required addressing several infrastructure constraints, particularly related to container image availability and rate limiting policies. Initial deployment attempts using standard Docker Hub repositories encountered rate limiting issues that prevented successful automated installation in the CI environment. The solution involved configuring the noobaa-operator to utilize CERN’s internal container registry, which provides reliable access to required container images without external rate limiting constraints: POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 7 CERN openlab Report 1/2025 # Install NooBaa with CERN registry (avoids Docker Hub rate limiting) ./noobaa-operator install --namespace=noobaa \ --noobaa-image='registry.cern.ch/docker.io/noobaa/noobaa-core:5.19.0'\ --operator-image='registry.cern.ch/docker.io/noobaa/noobaa-operator:5.19.0' # Configure NSFS Glacier backing store kubectl apply -f config/glacier-bucket-class.yaml kubectl apply -f config/nsfs-glacier-config.yaml Listing 1: NooBaa Installation Commands 3.2.2 The Configuration Challenge Getting NooBaa to work with our custom tape integration was like teaching a modern smartphone to work with a vintage stereo system – both are excellent at what they do, but they weren’t designed to work together. The NSFS Glacier configuration required some creative problem-solving. We had to patch NooBaa’s core configuration (its StatefulSet) to recognize our custom executables, and then carefully place our tools in exactly the right directories. It was like installing custom software on a device that wasn’t quite ready for it – one wrong step and the whole system would refuse to start. 3.3 Authentication 3.3.1 The Security Challenge We need a system that’s allow Noobaa to talk with CTA. We can request data to store or stage the archive files back to the disk from Noobaa itself. Therefore, # Generate regular user tokens for archival operations TOKEN_EOSUSER=$(eos root://${EOS_MGM_HOST}token --tree \ --path '/eos/ctaeos'--expires ${LATER}\ --owner user1 --group eosusers --permission rwx) # Generate poweruser tokens for retrieval operations TOKEN_POWERUSER=$(eospower_eos root://${EOS_MGM_HOST}token \ --tree --path '/eos/ctaeos'--expires ${LATER}) Listing 2: Authentication Token Generation 3.3.2 The Two-Key Token System To maintain security and proper access control, the integration uses a two-tier token system. Not all tape operations should have the same level of authorization: while archiving data can be performed with standard permissions, more sensitive operations such as file retrieval require elevated privileges. This separation ensures that only trusted users or services can perform high-impact actions, protecting the system from accidental misuse or unauthorized access. POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 8 CERN openlab Report 1/2025 5.1.1 Complete S3 Glacier API Compatibility •Full archival workflow: S3 PutObject with GLACIER storage class support •Complete retrieval workflow: S3 RestoreObject operations with tape staging •Status monitoring through standard S3 APIs and custom executable integration •Bulk operations supporting thousands of files with concurrent processing 5.1.2 Robust Archival Validation •EOS archiveinfo API polling implementation for definitive tape validation •"locality": "TAPE" status verification ensuring actual tape storage •Comprehensive error handling and retry mechanisms with exponential backoff •Configurable timeout handling (5-minute default with 10-second polling intervals) 5.1.3 Token-Based Authentication System •Dynamic EOS bearer token generation for API authentication •Differentiated token permissions for archival versus retrieval operations •Regular user tokens (user1:eosusers) for archival operations •Poweruser tokens (poweruser1:powerusers) for tape staging requests •Automated token lifecycle management and secure caching mechanisms 5.2 Performance Metrics Comprehensive testing demonstrated excellent system performance characteristics: •Archival Success Rate: 100% across single-file and bulk operations •Bulk Processing: Successfully validated with 100+ files •Retrieval Success: Complete end-to-end workflow from tape to client download 6 CHALLENGES AND SOLUTIONS 6.1 Technical Implementation Challenges 6.1.1 NooBaa NSFS Glacier Limitations Challenge: NooBaa’s native glacier command implementation incomplete ("TODO" status in codebase). Solution: Developed comprehensive custom executable suite to provide missing Phase 2 archival functionality. POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 15 CERN openlab Report 1/2025 6.1.2 Container Registry Rate Limiting Challenge: Docker Hub rate limiting severely impacting CI pipeline reliability. Solution: Implemented CERN pull-through cache using registry.cern.ch/docker.io/ prefix, eliminating external dependencies. 6.1.3 Archival Validation Complexity Challenge: Distinguishing between EOS disk buffer storage and actual tape archival completion. Solution: Implemented sophisticated archiveinfo API polling with "locality": "TAPE" status verification. 6.2 Authentication Architecture Challenges 6.2.1 Multi-Tier Permission Requirements Challenge: Different operations requiring distinct authorization levels for security compliance. Solution: Implemented differentiated token-based authentication with role-specific permissions for archival versus retrieval operations. 6.2.2 Token Lifecycle Management Challenge: Secure token generation, distribution, and lifecycle management across Kubernetes environments. Solution: Developed automated token generation and secure transfer mechanisms with proper expiration handling. 7 CONCLUSION The NooBaa-CTA integration project successfully delivers a prototype, S3 Glacier API interface for CERN Tape Archive systems. The implementation provides seamless tape storage access through industry-standard S3 APIs while preserving CTA’s robust tape management capabilities and operational reliability. Key technical achievements include complete S3 Glacier workflow implementation, comprehensive tape archival validation, large-scale bulk processing capabilities, and seamless CI/CD integration. The token-based authentication system provides secure, role-based access control while maintaining operational simplicity. The solution addresses a critical infrastructure need for S3 API compatibility in enterprise tape storage environments and establishes a solid architectural foundation for future scalability enhancements. The project demonstrates successful integration of modern cloud-native APIs with traditional enterprise tape infrastructure, enabling CERN’s scientific computing POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 16 CERN openlab Report 1/2025 workflows to leverage both contemporary application patterns and proven enterprise-grade tape storage systems. This work provides significant value for organizations seeking to modernize tape storage access while maintaining existing investment in tape infrastructure, offering a practical bridge between legacy storage systems and cloud-native application architectures. 8 References •https://www.noobaa.io/ •https://eoscta.docs.cern.ch/latest/ POC ON-PREM IMPLEMENTATION OF S3 WITH COLD STORAGE LAYER 17