Connecting REANA and the CERN-VRE: a JupyterLab Extension Middleware

Connecting REANA and the CERN-VRE: a JupyterLab extension middleware AUGUST 2024 AUTHOR(S): Rubén Pérez Mercado University of Granada SUPERVISOR(S): Enrique García García (IT-GOV-ENG) Giovanni Guerrieri (IT-GOV-ENG) CERN openlab Report // 2024 2 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware ABSTRACT In modern scientific research, preserving and reproducing analyses and results is crucial. The data gathered by experiments is shared among international collaborations and is analysed with software in continuous development, complicating the tasks of maintaining consistency and robustness. Within the ESCAPE and EOSC Future projects, an analysis facility was developed using a collaborative, bottom-up approach between all project partners. This initiative led to the CERN Virtual Research Environment, a platform that integrates several key components to streamline data analysis. It provides access to the ESCAPE Data Lake infrastructure through the CERN-developed data management framework (Rucio), and incorporates REANA – an open-source tool designed by CERN that focuses on ensuring result reproducibility and facilitating re-analysis. This contribution introduces Reana JupyterLab, a JupyterLab extension that integrates REANA into any JupyterHub-based environment, such as the CERN Virtual Research Environment. This extension allows the users to connect to any REANA instance, display their workflows, run new analysis and download data and results to the remote environment. The accessibility for non-expert users is ensured by providing a plug-and-play graphical interfaces. CERN openlab Report // 2024 3 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware TABLE OF CONTENTS INTRODUCTION 04 THE VIRTUAL RESEARCH ENVIRONMENT REANA DESCRIPTION OF THE PROJECT 04 PROBLEM MAIN FEATURES IMPLEMENTATION CONCLUSIONS 08 REFERENCES 09 CERN openlab Report // 2024 4 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware 1. INTRODUCTION In scientific research, the complexity and scale of data analysis have grown exponentially. Researchers across various disciplines are required to handle large datasets, perform complex analyses, and ensure the reproducibility of their results. However, most of the traditional facilities fail to provide a unified analysis framework, forcing scientists to exploit different resources across various platforms or services. This is where the Virtual Research Environment (VRE) comes into play. a. THE VIRTUAL RESEARCH ENVIRONMENT The VRE is a Jupyter-based collaborative analysis platform where researchers from different scientific communities can develop and share end-to-end workflows, having access to all the digital content needed to produce a scientific result [1][2]. It is developed at CERN in the context of the ESCAPE and the European Open Science Cloud Future projects, and its main objective is to establish a common set of tools and frameworks that physicists from different areas can use. This will foster collaboration across scientific groups, as well as build a shared community of developers and operators. The platform consists of four main components: • An Authentication and Authorization layer based on INDIGO IAM service [3]. • The ESCAPE Data Lake Infrastructure, that uses Rucio [4] as the data management framework. • A Reana cluster [5]. • An enhanced notebook service (JupyterHub interface). It is worth noting that one of the enhancements is a Rucio plugin that enables the user to interact with the Data Lake 1 . b. REANA REANA [5] is a reusable and reproducible research data analysis platform, born to facilitate code and data reuse. The platform aims to facilitate the adoption of declarative workflow systems to run data analysis processes on remote compute clouds. All the steps of analyses are defined in a single YAML file, containing the inputs (i.e., data files, parameters and live database calls), the software (e.g., code, frameworks), the environment (e.g., operating systems, CPU and memory resources) used to run the analysis, and the computational steps of the pipeline (e.g., shell commands, local or remote task executions). With this specification, the REANA platform can start the analysis and provide an output. 2. DESCRIPTION OF THE PROJECT a. PROBLEM REANA can be used through a Python library called reana-client 2 . This library allows the interaction with REANA via CLI or Python scripts. A few years ago, a web-client (reana-ui) was created as part of a previous Openlab project, providing a graphical interface that facilitated the interaction with the service [6]. However, if a researcher wants to interact with REANA within a JupyterLab interface, the only way to do that is by using reana-client. It is possible to use reana-ui, but it will be necessary to open a 1 https://github.com/rucio/jupyterlab-extension 2 https://github.com/reanahub/reana-client CERN openlab Report // 2024 5 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware new browser tab. Therefore, the objective of this project is to create a JupyterLab extension, similar to the aforementioned Rucio plugin, that integrates REANA and its functionalities in the Jupyter-based Analysis Facility (AF). b. MAIN FEATURES The JupyterLab extension allows the user to perform all the operations that are available in reana-ui and create new workflows from the environment. The following sections describe all the features implemented. Connect to any REANA server When opening the extension for the first time, the user needs to enter the REANA credentials in the platform (i.e., the server URL and the access token). For this reason, the only enabled tab in the extension is Connect, which contains a form in which the user must enter the URL of the REANA server and the access token the user has for that server (Figure 1). Once the user enters the credentials and clicks on the Connect button, a notification happens to appear in the bottom right corner of the screen indicating that the connection was successful. The other tabs are then enabled, allowing the user to interact with the REANA server. Figure 1. ‘Connect’ tab. Connection form. CERN openlab Report // 2024 6 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware List workflows After connecting to the server, a tab named Workflows is activated (Figure 2). This tab allows the user to display the list of workflows. The list includes information about each workflow (i.e., the name, the run number and the status). The user can interact with the workflows by clicking on any of them. There is also a filter area available that allows the user to customise the search by name and by status. The results can also be ordered by the creation date (in ascending or descending order). A refresh button is present so that the user can update the list at any moment. Figure 2. ‘Workflows’ tab. List of workflows. Display the details of a workflow Should a user click on any of the workflows in the previous list, the details of the workflow appear (Figure 3). In addition to the name, the run number and the status, the extension also displays additional information like the number of steps and the date. Furthermore, the user finds a menu bar with four different options: • Engine logs: Displays the engine logs of the workflow, which includes information about the execution of the workflow. • Job logs: Displays the job logs and information about each workflow step. The user can use a dropdown menu to select the step they want to see the logs of. CERN openlab Report // 2024 7 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware • Workspace: Displays the workspace related to the current workflow. The user can browse the workspace by name using the search bar. It is possible to download the whole workspace to the virtual environment by clicking on the download button next to the search bar. Rather than the whole workspace, individual files or groups of them can be selected for download as well. After the download, a notification appears at the bottom right of the screen with the number of files downloaded, and a new directory with the name and the run of the workflow appears in the root directory of the virtual environment containing the downloaded files. • Specification: Displays the specification and runtime parameters of the workflow. Figure 3. ‘Workflows’ tab. Workflow details and job logs. Create a new workflow from the virtual environment One of the features that reana-ui does not implement is the creation of a new workflow from the local or remote environment. This functionality is implemented within the extension (Figure 4). In order to create a workflow, the user must enter a workflow name and a YAML file. The name does not need to be unique (if it was already used, it will be considered a new run of the workflow), and it must only contain alphanumeric characters and underscores. For the YAML file, a file browser is available. Optionally, the YAML file can be validated by clicking on the Validate button. The extension checks the syntax of the YAML file and displays the output on the screen. If the validators report an error, the message is displayed in a red box, otherwise, it is displayed in a blue box. After selecting the name and the YAML file, the user can click on the Create & Run button. This button automatically creates the workflow, uploads the input files to the REANA workspace and starts the pipeline. A message is displayed indicating whether the workflow was created successfully. As the user navigates to the Workflows tab, the new workflow appears in the list. CERN openlab Report // 2024 8 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware Figure 4. ‘Create’ tab. Creation form with the output after validating the YAML file. c. IMPLEMENTATION Reana JupyterLab 3 is composed of two main components: the frontend and the backend. The frontend is a JupyterLab extension coded in TypeScript and React. The backend is a JupyterLab server extension coded in Python. The frontend is responsible for the user interface and the backend is responsible for the server-side logic. The backend is connected to a REANA cluster by using the REANA REST API in almost all the operations. It is only in the creation and validation actions where the commands provided by reanaclient are used, due to the lack of relevant API endpoints. However, the idea was to use the Python methods available in reana-client. This option was discarded because some of the methods were not working as expected. The package is present in PyPI as reana-jupyterlab 4 . A Docker image with all the software needed to use this extension 5 is also available in the GitHub repository page. 3. CONCLUSIONS The Reana JupyterLab extension is a significant addition to the VRE project, integrating the REANA workflow management system in any JupyterHub-based service (e.g., VRE or ideally SWAN in the near future). It provides a user-friendly platform that enables researchers to develop, execute and analyse scientific workflows within a unified environment, enhancing collaboration and reproducibility in research. 3 https://github.com/vre-hub/reana-jupyterlab-extension 4 https://pypi.org/project/reana-jupyterlab/ 5 https://github.com/vre-hub/reana-jupyterlab-extension/pkgs/container/reana-jupyterlab-extension CERN openlab Report // 2024 9 Connecting REANA and the CERN-VRE: a JupyterLab extension middleware 4. REFERENCES [1] Gazzarrini, E., Garcia, E., Gosein, D., Moya, A. V., Kounelis, A., & Espinal, X. (2023). The Virtual Research Environment: towards a comprehensive analysis platform. arXiv preprint arXiv:2305.10166. [2] Gazzarrini, E., Garcia, E. G., Gosein, D., & Espinal, X. (2024). The Virtual Research Environment: A multi-science analysis platform. In EPJ Web of Conferences (Vol. 295, p. 08023). EDP Sciences. [3] Ceccanti, A., Hardt, M., Wegh, B., Millar, A. P., Caberletti, M., Vianello, E., & Licehammer, S. (2017, October). The INDIGO-Datacloud authentication and authorization infrastructure. In Journal of Physics: Conference Series (Vol. 898, No. 10, p. 102016). IOP Publishing. [4] Barisits, M., Beermann, T., Berghaus, F., Bockelman, B., Bogado, J., Cameron, D., ... & Wegner, T. (2019). Rucio: Scientific data management. Computing and Software for Big Science, 3, 1-19. [5] Šimko, T., Heinrich, L., Hirvonsalo, H., Kousidis, D., & Rodríguez, D. (2019). REANA: A system for reusable research data analyses. In EPJ Web of Conferences (Vol. 214, p. 06034). EDP Sciences. [6] Sinclert Pérez. (2018). User dashboard for reusable analysis platform. Zenodo. https://doi.org/10.5281/zenodo.2415739