scieee AI-readable full text Open interactive document viewer

A Data Science Platform for the National Oceanography Centre and Beyond

Sauze, Colin; Turner, Esther

Publisher
Zenodo
DOI
10.5281/zenodo.17967460
Repository
zenodo.org
Language
en

Abstract

One of the common issues faced in the world of research software is "it worked on my machine". In this talk, we will present our efforts at the National Oceanography Centre (NOC) to solve this problem and make the task of writing shareable and reusable software easier by building an interactive Data Science Platform (DSP) based upon JupyterHub. The platform is accessed through a web browser and includes the Pangeo [https://pangeo.io/] stack along with popular machine learning libraries. In addition to a Jupyter notebooks interface we have integrated Matlab and a remote desktop service, removing the need for users to install these locally or rely on X forwarding/NoMachine/VNC. The DSP was first deployed on servers owned by NOC. We are employing cloud native technologies such as Kubernetes and S3 style object stores as the basis for much of our digital research infrastructure. This allows us to deploy instances of the DSP on commercial cloud and national computing facilities, providing a uniform user experience regardless of the underlying system. Where the DSP’s server is not powerful enough, we use Dask to enable access to high performance computing or cloud computing resources. In addition to providing a resource for developing and running research code, the platform is ideal for training. We use it regularly for Carpentries workshops since it provides all the software the learners require, resulting in a significant time saving during workshops, and allows ongoing access after the workshop. As well as a demonstration of this platform, we will present our experiences of deploying and running it, including lessons we have learnt along the way and our plans for its future. Acknowledgements This work has been funded by the Natural Environment Research Council (NERC) projects Atlantic Climate and Environment Strategic Science (AtlantiS) and Climate Linked Atlantic Sector Science (CLASS).A recording of this session is available on YouTube: https://youtu.be/ovKPLFRQwdw

Full text

A DATA SCIENCE PLATFORM FOR THE NATIONAL OCEANOGRAPHY CENTRE AND BEYOND 09 September 2025 Colin Sauze, Esther Turner WHY ARE WE BUILDING A DATA SCIENCE PLATFORM? HAVE YOU HEARD THIS BEFORE? "It worked on my computer!" "I have to leave my laptop on overnight because my script is still running..." "I don't know how to install [software] on my computer." "I don't have a GPU!" "NoMachine is slow" •Aiming to transform the organisation through investment in and development of: oDigital Skills oDigital Infrastructure oAI and ML Capabilities oExternal Partnerships •Data Science Platform underpins this: oEasier to access oSupports training oAccelerated hardware for AI/ML •Initial audience are the ~400 researchers at NOC •Test deployment since summer 2024 oGradual increase in users, with over 100 now registered DIGITAL TRANSFORMATION AT NOC WHAT IS IT? •Accessed via a web browser •No need to install anything on your laptop JUPYTERHUB BASED PLATFORM Below: A screenshot of JupyterHub running a notebook to plot global sea surface temperatures Above: A basic architecture diagram for the Data Science Platform •Default environment based on Pangeo with machine learning dependencies (PyTorch, TensorFlow, Keras) •Common Linux terminal utilities including Git, full man pages, vi(m)/nano/emacs, make/gcc •Other environments prepared for projects and workshops •Dropdown menu of Docker images •Most users won't have to worry about configuring environments themselves PRE-CONFIGURED SOFTWARE ENVIRONMENTS Above: A screenshot of a Pangeo webpage showing some of the libraries included: Xarray, dask, Zarr, Jupyter, hvPlot, Kerchunk, Intake, xbatcher, VirtualiZarr, XPublish, Cubes and Xarray-Beam •Available through a notebook with a Matlab kernel oEase the transition to Python •Or through a Matlab web interface •Using NOC's license server (not possible to replicate this in deployments without a license) INTEGRATED MATLAB Left: Screenshot of the Matlab native style interface. Top: Screenshot of the Matlab Jupyter interface. •A selection of light weight remote desktops available (LXDE as default + XFCE, JWM and OpenBox) •Enables graphical Linux applications to be run in a web browser •Can swap computers and pick up where you left off •Faster than NoMachine •No client to install REMOTE DESKTOP Above: Screenshot of QGIS running on the Remote Desktop TRAINING WORKSHOPS •Open-source workshops for scientists using software engineering and data science materials from The Carpentries •No installation of software required •Large time saving at the start of lessons! oEstimated 30-60 minutes saved on larger workshops •Spent a long time downloading packages during a Conda workshop oconda-forge doesn't seem very fast for us o1 gigabit/second network link doesn't help when you have 20+ learners trying to download packages oWe need a local conda-forge mirror for Conda workshop Similarly, project workshops: Enable demos of project software to large groups (including external stakeholders) without worrying about different set-ups. WHAT LESSONS HAVE WE LEARNT? LESSONS LEARNT A screenshot of the Munin monitoring system. •Initial deployment summer 2024 oWe now have one year of seeing how the platform is used •You can pre-install many things but not everything oThere will be contradictory requirements •Don't offer too many choices oLet power users configure things themselves •Give users access to system performance information oWe currently have no memory, CPU or GPU limits WHAT ARE OUR FUTURE PLANS? •HPC Connection with Dask oDask backend to connect to HPCs ▪Easy access to HPC from Jupyter/Xarray •Data Browsing oWidget to browse our data catalogue •Rocky/LMod oBuilding an image based on Rocky Linux which can use our existing catalogue of loadable Lmod modules •SSH Interface oOur power users don't want to use a web interface oThey want SSH access to a container •Examples Gallery oCommon examples people can base new work on oCreating an example from every (relevant) project we work on FUTURE MODIFICATIONS Above: A screenshot of the current examples gallery for the DSP, containing three notebooks •We are going to buy a new Kubernetes cluster(s) oBig enough for all our users ▪Also used for running other web applications used across the organisation oBetter GPUs ▪Possibly NVIDIA L40s, about 10x faster than a P100, 3x memory increase oGradually replacing our shared Linux servers oOne cluster for each of our (on land) sites ▪Ship based severs a consideration in the longer term NEW ON-PREMISES HARDWARE TIME FOR A DEMO •MyBinder demo with a lightweight version of the image ohttps://github.com/NOC-OI/dsp-binder-demo DEMO WHAT ABOUT INTERACTIVE GRAPHICAL SOFTWARE? CAN IT RUN DOOM? HTTPS://CANITRUNDOOM.ORG/