Architecting the Datastore for the URL Frontier of OpenWebSearch.eu

ARCHITECTING THE DATASTORE FOR THE URL FRONTIER OF OPENWEBSEARCH.EU Noor A. Fathima∗1, M. Dinzinger2, M. Granitzer2, A. Wagner1 1CERN, Geneva, Switzerland 2University of Passau, Passau, Germany Abstract This paper presents the architectural evolution of the URL Frontier datastore within the OpenWebSearch.eu initiative [2], transitioning from an OpenSearch-based prototype [3] to a high-throughput ScyllaDB deployment [4]. Motivated by the need for low-latency, write-optimized infrastructure to support continuous web crawling, we conducted a structured evaluation comparing OpenSearch, HBase [6], Cassandra [7], and ScyllaDB across performance, scalability, operational complexity, and infrastructure compatibility. Our findings identified ScyllaDB as the most suitable datastore due to its shard-per-core design using the Seastar framework [8], SSD optimization, and minimal maintenance overhead. We detail the deployment process using rootless containers managed via Podman [9] and secured through Puppet-managed nftables [10,11], as well as the integration with Scylla Manager for future cluster scaling [5]. BACKGROUND AND PRIOR WORK The OpenWebSearch.eu project originally adopted OpenSearch [3] as the primary back-end datastore for its URL Frontier service, based on the reference implementation from the URL Frontier project [12]. This decision was guided by the need for a system that could support document indexing, querying, and distributed replication with minimal integration overhead. OpenSearch, being a wellsupported fork of Elasticsearch, offered a mature ecosystem with open-source plugins, horizontal scalability, and realtime observability features that aligned with the project’s early priorities, especially in monitoring and debugging the crawling process. However, as crawling operations matured and scaled, limitations in OpenSearch began to surface. The URL Frontier, by nature, is a write-heavy system: URLs are rapidly inserted, updated, dequeued, and reprioritized as part of continuous crawling workflows. While OpenSearch performs well under read-intensive or search-centric workloads, it is not specifically optimized for high-throughput, low-latency write operations, particularly when frequent document updates are involved. This mismatch began to manifest in performance bottlenecks and operational complexity as the size of the index grew and the system moved closer to productionscale loads. In response to these constraints, Apache HBase was proposed by a collaborator as a potential alternative. HBase [6], ∗noor.afshan.f[email protected] a column-family NoSQL database built on top of HDFS, is known for its suitability in high-ingestion, high-availability applications, particularly those requiring fast random writes and structured row-based access. In internal communications, it was noted that HBase’s data model better aligns with the URL Frontier’s query pattern, especially for managing state transitions and batch updates. The proposed deployment architecture mirrored the traditional HBase setup: a single Zookeeper node for coordination, one master, and multiple RegionServers for data handling. However, a deeper evaluation of HBase revealed several operational challenges that made it incompatible with our existing infrastructure. Our environment is based on OpenStack-managed bare-metal servers with SSD storage and S3-compatible object storage—neither of which are natively compatible with the HDFS backbone required by HBase. Setting up and maintaining an HDFS cluster would introduce considerable overhead and complexity, particularly without an existing Hadoop ecosystem. While adaptations of HBase for object storage do exist, they typically come with reduced performance and limited community support, further disincentivizing this route. As a result, we expanded our exploration to alternative NoSQL databases that could better utilize SSDs, integrate seamlessly with our infrastructure, and support the writeheavy nature of the Frontier workload. This led us to evaluate ScyllaDB [4], a drop-in replacement for Cassandra [7], but reengineered in C++ to deliver lower latency and better throughput on modern multi-core hardware. The following sections will outline how ScyllaDB emerged as the preferred solution and how its architecture better aligns with the operational and performance requirements of the URL Frontier at scale. EVALUATION FRAMEWORK AND DESIGN CRITERIA To guide the evaluation of candidate datastores for the URL Frontier, we conducted a structured diagnostic using a set of targeted questions across four domains: (1) Data Storage and Management, (2) Performance and Scalability, (3) Infrastructure and Resource Utilization, and (4) Data Consistency and Reliability. These questions helped us identify the practical limitations of our OpenSearch-based setup and assess alternatives like Apache HBase and ScyllaDB. The diagnostic framework provided clarity on aspects such as write performance https://doi.org/10.5281/zenodo.17246104 under load, SSD utilization, data partitioning strategies, and operational complexity in bare metal environments. Based on the outcomes of this evaluation—summarized in Appendix A 1 —we identified ScyllaDB as the most suitable candidate. It addressed the primary bottlenecks around writeheavy operations and infrastructure alignment while offering improved predictability and observability for future scaling. The diagnostic evaluation outlined above guided a comparative analysis of four candidate systems: ScyllaDB, OpenSearch, Cassandra, and HBase. The table1 below summarizes their key characteristics and how they align with our system’s architectural, operational, and scalability requirements: Based on this comparison and further internal testing, ScyllaDB emerged as the optimal datastore for the URL Frontier service. The next section details how its design aligns with our infrastructure and performance goals. DESIGN MOTIVATION AND SYSTEM SELECTION ScyllaDB is specifically designed to take full advantage of modern multi-core servers like the one used in our infrastructure (64-core, 256 GB RAM, 12 TB SSD). Its architectural features align closely with the requirements of the URL Frontier service: Shard-per-Core Architecture ScyllaDB operates using a shard-per-core architecture, creating 64 independent shards on our 64-core machine. Each shard is assigned to a dedicated core and handles a subset of the data. This isolation minimizes cache contention and cross-CPU communication, enabling high parallelism and efficient core utilization. Memory Management The 256 GB of RAM is partitioned across shards, allowing each to manage its memory independently. This enables efficient row-based caching and minimizes disk I/O. Unlike Java-based systems, ScyllaDB’s C++ implementation avoids garbage collection pauses, leading to more consistent performance. I/O Optimization ScyllaDB employs an I/O scheduler tailored to SSDs and leverages the Seastar asynchronous framework. Each shard conducts non-blocking I/O operations directly, capitalizing on the high IOPS of SSDs to ensure fast read/write operations. Networking and Client Requests ScyllaDB’s shard-aware drivers route client requests directly to the relevant shard, reducing request overhead and improving latency. This design allows balanced load distribution across all cores. 1 A full list of the evaluation questions and their corresponding answers can be found in Appendix A. Thread and Storage Management Each core runs a single thread, avoiding context-switching overhead. This ensures predictable latency and high throughput. With direct SSD access, each shard manages its own data, taking advantage of parallelism at both the compute and storage layers. Benefits for Our Setup • High Throughput: The architecture fully utilizes our 64-core machine for parallel processing. • Low Latency: Combined effects of SSDs, shard-local caching, and async I/O provide rapid data access. • Efficient Scaling: The system supports future growth in data volume while maintaining performance. In summary, ScyllaDB’s architectural principles and implementation make it a natural fit for our infrastructure and the demanding requirements of the URL Frontier. The selection was informed by a combination of empirical evaluation, expert advice, and practical deployment considerations. The following section will describe the deployment process and operational setup in more detail. DEPLOYMENT AND OPERATIONAL CONSIDERATIONS Initial Setup ScyllaDB was installed and deployed on the dedicated URL Frontier server, a 64-core, SSD-equipped bare-metal node provisioned via CERN’s OpenStack infrastructure. The deployment was configured for single-node operation initially, with horizontal scaling planned via Scylla Manager. Networking and Security Network access was managed using Puppet to enforce firewall rules with nftables. The local ruleset allowed containerlevel access on the CQL port (9201), while requests for perimeter-level access were submitted to the CERN Security team. The security group was updated to allow external services to connect to the ScyllaDB instance without exposing unnecessary surfaces. Installation Method Although the official documentation recommends direct installation on the host for maximum performance, we opted for a containerized deployment using Podman. This choice balanced maintainability, isolation, and consistency across environments with minimal observed performance overhead. The container was managed as a rootless systemd service, ensuring automatic start-up and persistent state across reboots. Cluster Scaling and Future Setup Plans are underway to expand the deployment into a multinode Scylla cluster using Scylla Manager, which requires its own metadata store (own ScyllaDB instance) and an agent on each node. This setup will enable automated repair, backup, https://doi.org/10.5281/zenodo.17246104 Table 1: Comparison of Datastore Candidates for the URL Frontier Feature ScyllaDB OpenSearch Cassandra HBase Architecture Shard-per-core, shared-nothing, based on Seastar Distributed, document-oriented Peer-to-peer, masterless Master-slave, columnfamily on HDFS Programming Language C++ Java-based Java-based Java-based Performance High throughput, low latency, optimized for SSDs Good for read-heavy and search operations High write throughput, some latency from GC High write throughput, higher latency from HDFS Scalability Linear scaling, efficient use of modern CPUs Scales with careful shard management Linear scaling, good for large-scale data Scales well but requires HDFS and Zookeeper Latency Low latency due to C++ and direct core access Low for read-heavy tasks, may struggle with writes Low to moderate, affected by GC under load Moderate, higher due to HDFS Shard Management Automatic, efficient shard-per-core utilization Requires careful management to optimize Automatic partitioning, less fine-grained Managed with RegionServers Caching Built-in row-based cache, no external cache needed External or built-in cache strategies needed Key and row cache; external cache often needed Block cache, may need external caching Data Consistency Tunable consistency Eventual consistency, strong for specific configs Tunable consistency Strong consistency by default Operational Complexity Minimal, selfoptimizing schedulers Moderate, requires JVM tuning, shard adjustments Requires JVM tuning, careful GC management High, due to HDFS, Zookeeper, and RegionServers I/O Optimization Custom I/O schedulers for storage types Heavily disk I/O reliant, JVM-based Decent I/O handling, but Java limits optimization Relies on HDFS I/O management Use Case Fit Real-time data ingestion, time-series, analytics Full-text search, log analysis, data visualization Fault-tolerant, IoT, distributed data Bulk analytics, structured data, strong consistency Resource Efficiency High, avoids GC issues, direct hardware use Moderate, JVM adds overhead Moderate, tuning required for optimal use Moderate, high overhead due to HDFS and Java Global Distribution Built-in multi-region support Can be configured for multi-region Good for georeplication Multi-datacenter possible but more complex Administrative Effort Low, minimal supervision needed Moderate, needs regular tuning Moderate, needs oversight for large clusters High, needs dedicated management monitoring, and performance insights. Integration and deployment of Scylla Manager are ongoing, with initial tests showing promise for future scaling and observability improvements. Results and Early Observations While comprehensive performance benchmarks are ongoing, early operational feedback has been highly positive. The transition to ScyllaDB significantly simplified maintenance tasks thanks to its self-tuning architecture and shard-aware design. Integration with the OWLer crawling stack was smooth, and the new datastore operates without disrupting the write-heavy demands of the Frontier service. Initial observations indicate that throughput and latency remain stable under expected load, with no degradation during integration or live crawling sessions. Future benchmarking will provide further insights into CPU utilization, storage IOPS, and overall system resilience. Future Work and Roadmap Several improvements are planned to extend the capabilities of the current datastore setup: • Full Cluster Expansion: Transition from a single-node deployment to a production-grade multi-node ScyllaDB https://doi.org/10.5281/zenodo.17246104 cluster. This will include the integration of Scylla Manager agents and a high-availability PostgreSQL metadata store. • Automated Observability: Incorporate Prometheus and Grafana dashboards for real-time monitoring of key performance metrics, including CQL latency, disk I/O, and memory usage per shard. • Resilience Testing: Implement fault injection tests to validate the resilience of the distributed ScyllaDB setup under node failures and high load. • CI/CD Integration: Automate the deployment and testing of ScyllaDB containers using GitLab CI/CD pipelines to streamline updates and ensure consistency across environments. • TTL and Retention Policies: Introduce automated TTL mechanisms for stale URLs and implement long-term storage policies to manage dataset growth efficiently. • Dual Store Design: Explore hybrid setups where ScyllaDB handles real-time operations, while OpenSearch remains active as a searchable store for crawl metadata and indexing. • Benchmarking Suite: Finalize a reproducible benchmarking suite to evaluate system performance under varied workloads and to inform scaling decisions moving forward. These enhancements aim to ensure long-term sustainability, performance, and transparency in managing the URL Frontier as the OpenWebSearch.eu infrastructure continues to scale. CONCLUSION This paper has presented the architectural journey behind redesigning the datastore for the URL Frontier of OpenWebSearch.eu, from its initial OpenSearch-based setup to a more scalable and write-optimized ScyllaDB deployment. Through a structured evaluation framework, practical testing, and operational integration, ScyllaDB emerged as a strong fit for the project’s evolving requirements. The migration has already yielded early benefits, particularly in system maintainability and seamless integration with the crawling stack. As we move forward, ongoing improvements in scalability, observability, and resilience will further strengthen the infrastructure’s role in enabling open, distributed, and ethically governed web search. By openly documenting these design decisions and tradeoffs, we aim to support broader efforts toward building transparent and community-driven alternatives to commercial web search infrastructure. ACKNOWLEDGEMENTS This publication has received funding from the European Union’s Horizon [1] Europe Research and Innovation Program under Grant Agreement No. 101070014 (OpenWebSearch.EU, https://doi.org/10.3030/101070014) APPENDIX A: EVALUATION STUDY OF ALTERNATIVE DATASTORES Following the non-adoption of the horizontally scaled OpenSearch cluster at CERN and subsequent requests to evaluate an alternative data store for the Frontier Service backend, this study outlines the assessment of potential replacements for OpenSearch. The initial decision to scale OpenSearch from a single node to a multi-node cluster was driven by the scope of the project, which aimed to avoid significant engineering efforts required for experimentation and deploying alternative systems. Comprehensive optimization of the OpenSearch cluster was performed, including fine-tuning configurations and leveraging CERN’s substantial hardware resources to mitigate bottlenecks. This approach was deemed sufficient within the project’s timeline and scope. However, due to the cited reasons that the data model of OpenSearch does not align well with the query pattern of the Frontier application and that it would remain a potential bottleneck despite horizontal scaling efforts, this assessment aims to explore alternative data store solutions. The goal is to identify a data store capable of sustaining and achieving the projected 10 TB per day crawling target while maintaining minimal operational complexity, high throughput, and low latency. The focus is on selecting a backend solution that meets the Frontier Application’s performance, scalability, and reliability requirements without incurring disproportionate engineering effort for deployment and ongoing maintenance. Earlier, CassandraDB was tested by the team members but was not adopted. There is a proposal to consider HBase as an alternative, based on the assertion that its data model better aligns with the query pattern of the Frontier application. To evaluate this proposal effectively, we will analyze it using the following key assessment points, which also help with our documentation and report writing tasks. Key Areas for Assessment: Understanding the Query Pattern of the Frontier Application Data Volume and Storage Needs Performance and Latency Requirements Scalability and Distribution Capabilities Consistency and Reliability Operational Complexity and Maintenance Caching and Performance Optimization Current and expected data model Partition Key Selection The questions related to each of the above-mentioned key areas of assessment are added as comments in this GL issue to keep the conversation organized. Please add the answers as corresponding comments. Conclusion: Selecting a suitable alternative to OpenSearch for the Frontier Service backend requires a thorough understanding of the application’s data handling patterns, performance expectations, and scalability needs. Addressing these questions will enable an informed assessment and help identify the data store that best meets the service’s requirements. https://doi.org/10.5281/zenodo.17246104 1. Understanding the Query Pattern of the Frontier Application Question: What is the detailed query pattern of the Frontier Application? Context: This will help determine whether the application primarily requires write-heavy operations, read-heavy operations, or a balance of both. Additionally, it will clarify if the application relies on sequential reads/writes, complex queries, or real-time data processing. Action: Document the typical query patterns, including examples of common read and write operations and their frequency. Response: The application is both read-heavy and writeheavy. However, the spectrum of queries is small. It basically breaks down to only three kinds of queries. The backend persists the crawl space: this is a large set of URLs, identified by an URL ID. This URL ID is a hash of the normalized URL, hence every URL is uniquely identified by the hexadecimal string representation of this SHA-256 hash. Besides that, meta information (as map/dictionary) is stored alongside each URL ID and URL. Most important metadata field is nextFetchDate , which specifies the timestamp of the next planned fetch. Additionally, the metadata map also contains a set of tags (like HTML, Adult, etc.), which impact the scheduling of URLs for crawling. To put it in a nutshell, the elements of the crawl space are uniquely identified by the URL ID and comprise several columns, namely URL, nextFetchDate and a static list of metadata fields. For the sake of distributing the crawl space among crawlers, the URL Frontier application divides the crawl space along the URL ID in 512 subsets/batches. The query pattern looks as follows (three kinds of queries): • Scan operation over a subset of the crawl space: This is a search request to retrieve new URLs to be fetched. The Frontier application scans over one subset/batch of the crawl space, which is ordered by the URL ID, and looks for all elements that meet certain filter criteria. The default filter criteria are: nextFetchDate has to be in the past (hence it is scheduled for crawling) and the element is tagged as HTML. • Exists operation for a set of URLs: After crawling, the crawler logs return the URL as well as discovered outlinks. For integrating these potentially new links to the crawl space, the Frontier application computes the URL ID for all these links and looks up whether these IDs are already persisted in the crawl space. Outlinks that have already been discovered can be thrown away and it is no expensive update operation necessary. • Update of crawled URLs and Insert of new discovered links: The update/insert operations are most expensive among the three kinds of queries in the query pattern. Crawled URLs, which are already in the crawl space, are updated with a new nextFetchDate and refined meta information. Discovered links are inserted with a nextFetchDate in the near future and a default set of meta information. Follow-Up Question: Is the application relying on random read/writes or sequential read/writes? Are the updates largescale modifications or incremental changes? Given that the nextFetchDate determines how frequently data is updated, how is nextFetchDate determined? Response: Read/write pattern: The read operations are sequential with respect to the URL ID, thus it is a Scan over the crawl space, which is ordered by URL ID, retrieving new URLs to be crawled next. The write operations are completely random. Scale of updates: The updates on the Frontier applications are ongoing, in order to extend our crawling from StormCrawler-only to more crawlers. In the Deliverable D1.2, I framed it as "Stream-based processing" (StormCrawler) and "Batch-based processing" (others). When implementing the Batch-based processing, one aspect became clear to me, which will—hopefully—increase performance significantly. The backend has to handle both read-heavy and writeheavy querying. Hence, the URLs have to be persisted in a shallow way, as an ordered list with a hash-based identifier. Consequently, all read operations can be sequential, and furthermore the number of write operations can be decreased by employing exists operations plus cheap insert operations instead of expensive update operations. OpenSearch actually allows cheap exists operations, yet it has two shortcomings: It is made for Search requests, but not for Scans. These require pagination; a lot of data has to be loaded into RAM, which makes these sequential reads expensive. Beyond that, the latency of write operations seems to not scale well, as it is directly dependent on the underlying index structures. I hope that HBase behaves differently in this regard. By definition, it only has a single index (like e.g. Cassandra, as well) and thus data access and (data manipulation) should be realized in a simpler, yet more performant way. How is nextFetchDate determined? It is determined by a software component of the URLFrontier called Scheduler. Basically, it adds two weeks to the current time, plus some minor adaptations to prioritize certain content (if the web page content has changed since the last crawl, the nextFetchDate is sooner; if it is Adult content, it is later, etc.) Clarification Request: When updates are performed on the Frontier Application, are they typically large-scale modifications affecting multiple records or incremental changes affecting individual records? https://doi.org/10.5281/zenodo.17246104 If I have understood correctly based on the answer to the next question too, then: • Since the nextFetchDate is updated on an ongoing basis, with adjustments influenced by real-time changes to the content and other factors, this implies that updates are generally incremental rather than large-scale modifications. • The updates seem to be frequent and distributed over time, rather than occurring in periodic bulk updates, due to the dynamic nature of scheduling based on content changes and prioritization rules. Response: Yes, every update comprises incremental changes. So every write operation updates the URL items that were crawled (so 1) and inserts newly discovered links (I try to keep this number small, so 0–8). However, one could collect these update operations as a bulk and send a bulk operation against the backend. This "bulking" could be implemented in custom logic (for HBase, I haven’t intended to do this so far) or, for the case of OpenSearch, it is already implemented in the logic of the Java client library. 2. Data Volume and Storage Needs Question: What is the current and projected volume of data stored and processed per day? Context: Establishing data volume requirements is essential to ensure that the new data store can handle current loads and scale efficiently to be able to crawl 10 TB/day. Action: Review current data metrics and perform projections based on growth patterns. 3. Performance and Latency Requirements Question: What are the current performance benchmarks (e.g., write latency, read latency) that must be maintained or improved? Context: Identifying performance metrics will help evaluate which data stores meet or exceed these benchmarks under similar or greater loads. Action: Analyze existing performance data and define acceptable latency thresholds for both read and write operations. Response: Update operations are most expensive and their latency has been the bottleneck in the OpenSearch setup. In optimal case, the costs of an Update operation should be agnostic to the size of the crawl space. However, for OpenSearch, as the index was growing, all read and write operations became more expensive. The write latency has been a bigger problem as the read latency, as the stream-based processing of the StormCrawler is unfortunately not robust enough to handle congestions resulting from slow updating of the crawl space. 4. Scalability and Distribution Capabilities Question: How well can the candidate data stores scale horizontally and distribute data across nodes? Context: The ability to scale without significant operational overhead or performance loss is crucial for supporting high-volume data ingestion and processing. Action: Compare the scalability characteristics of each alternative (e.g., linear scaling, horizontal scaling support) against OpenSearch. 5. Consistency and Reliability (urgent) Question: What level of consistency is required by the Frontier Application, and can the alternative data stores provide this? Context: The Frontier Application may have specific requirements for strong, eventual, or tunable consistency. The choice of data store should align with these needs to maintain data integrity. Action: Clarify the consistency model needed and assess each candidate’s ability to provide it, including their mechanisms for data replication and failure recovery. Response: I’d say the Frontier application requires strong consistency for update operations. After an update of a crawled URL with a new nextFetchDate, it has to be guaranteed that it will not be read with the old - now invalid - nextFetchDate again, meaning that it is crawled twice within a short period of time. In reality, eventual consistency would however probably be also okay as a scan over a subset/batch of the crawl space presumably takes longer than the eventual consistency to realize. For insert operations of newly discovered links, eventual consistency is okay. 6. Operational Complexity and Maintenance Question: What is the operational overhead associated with maintaining the data store? Context: Ease of maintenance, monitoring, and scaling is important to minimize downtime and manual intervention. Action: Evaluate the complexity of setting up, managing, and scaling each data store, considering factors like configuration, monitoring, and required expertise. 7. Caching and Performance Optimization Question: Does the data store have built-in caching mechanisms, or will it require external caching solutions to meet performance needs? Context: Caching capabilities can greatly impact the efficiency of read-heavy workloads and reduce latency. Action: Assess the need for built-in vs. external caching solutions for each candidate and their impact on overall performance. 8. Current and Necessary Data Model (not urgent) Question: What is the current and expected data model that is necessary? Context: Understanding the expected data model is crucial to determine whether HBase or any other alternative is a better fit for the Frontier application. This involves identifying how data is structured, accessed, and updated within the application. The analysis should focus on whether the data model aligns with the application’s query patterns, data relationships, and workload characteristics (e.g., write-heavy or read-heavy operations). Action Points: • Review and document the current data model used by the Frontier application. • Identify key data attributes and relationships critical for the application’s functionality. https://doi.org/10.5281/zenodo.17246104 • Analyze the compatibility of the expected data model with HBase and compare it with OpenSearch and other potential data stores. 9. Partition Key Selection (urgent) Question: What is the current partition key? Context: Understanding how the partition key was selected is vital for assessing the current system’s effectiveness and identifying potential performance bottlenecks or hotspots. The partition key plays a critical role in data distribution, load balancing, and system performance across nodes or shards. Analyzing the current partition key choice helps ensure data is evenly distributed, preventing nodes or regions from being overwhelmed. Action Points: • What partition key is currently being used for the Frontier Application, and why was this choice made? • Can you provide examples of how the current partition key affects data distribution across nodes or shards? • Have you observed any hotspots or imbalances in data distribution related to the chosen partition key? If so, what measures have been considered to address these? Response: The prior OpenSearch-based backend uses a hash of the PLD (Paid-Level Domain; domain) as partition key. The hash is an integer and further taken modulo NUM_BATCHES, which has been 400. So we have 400 partitions and each partition is one index (and each index has one shard, so 400 OpenSearch shards). The hash of the PLD is not perfectly evenly distributed, but close enough for our case. Plus it is ensured that URLs of the same domain are in the same partition. However, when moving to batch-based processing, this is not necessary anymore. The only identifier is the URL ID, and read operations are either sequential scans over the range of URL IDs or exists/get operations looking for single URL IDs. Data partitions can be arbitrary as long as URL IDs are persisted sequentially. There has not been any hotspots or imbalances in data distribution. REFERENCES [1] Horizon Europe, https://research-and-innovation. ec.europa.eu/funding/funding-opportunities/ funding-programmes-and-open-calls/ horizon-europe_en [2] OpenWebSearch.eu Consortium. OpenWebSearch Project Overview. [Online]. Available: https://openwebsearch. eu [3] OpenSearch Project. OpenSearch Documentation. [Online]. Available: https://opensearch.org/docs [4] ScyllaDB Inc. ScyllaDB Documentation. [Online]. Available: https://docs.scylladb.com [5] ScyllaDB Inc. Scylla Manager Overview. [Online]. Available: https://docs.scylladb.com/operating-scylla/ manager [6] George, L. HBase: The Definitive Guide. O’Reilly Media, Inc., 2011. [7] Lakshman, A., Malik, P. “Cassandra: A decentralized structured storage system,” in ACM SIGOPS Operating Systems Review, vol. 44, no. 2, pp. 35–40, 2010. [8] ScyllaDB Inc. “Seastar: High performance serverside application framework,” [Online]. Available: https://www.scylladb.com/2017/02/14/ seastar-high-performance-application-framework [9] Red Hat. Podman Documentation. [Online]. Available: https://podman.io [10] The Netfilter Project. nftables wiki. [Online]. Available: https://wiki.nftables.org [11] Puppet Inc. Puppet Documentation. [Online]. Available: https://puppet.com/docs [12] OpenSearch implementation of the URL Frontier https://github.com/PresearchOfficial/ opensearch-frontier/ https://doi.org/10.5281/zenodo.17246104