The Data Acquisition System for the complete KM3NeT/ARCA neutrino telescope
Abstract
Parallel talk presented at the XXI International Workshop on Neutrino Telescopes - Padova 29 September - 3 October 2025 (https://agenda.infn.it/event/44606/) On behalf of the KM3NeT Collaboration Abstract: The KM3NeT observatory hosts two undersea neutrino telescopes, ARCA and ORCA, located at two abyssal sites of the Mediterranean sea. The detectors consist of a 3D array of optical modules, each housing 31 3-inch photomultiplier tubes to detect Cherenkov light emitted by charged particles produced in neutrino interactions in water. Although still under construction, both detectors are already using the same data acquisition model in compliance with the triggerless-streaming readout approach. In this architecture all the data collected by optical modules are transmitted to shore, where online processes running on dedicated resources filter and record relevant data for physics analyses and calibration procedures. To accomplish the target scientific goals, stringent constraints on the precision of the position and timing of the modules are set. In particular the clock distribution must provide a nanosecond synchronisation of the modules which are tens of kilometers away from the on-shore clock references and occupy large volumes that, in the case of ARCA, can reach the cubic kilometer scale. This requirement is met by exploiting the White Rabbit technology.ORCA telescope and Phase-1 of the ARCA telescope, which accounts for 13% of the total instrumented detector, are based on a custom White Rabbit implementation that deviates significantly from the standard design. Recently the connection layout of the ARCA telescope was significantly revised to accomplish a mandatory optimisation necessary to scale it to the cubic kilometer size. In this new scenario, which is referred to as the Phase 2 of ARCA construction, it was possible to revise also the implementation of the White Rabbit technology, restoring the original standard design. In this contribution the evolution of the ARCA data acquisition system from Phase 1 to Phase 2 is reviewed, focusing on the revised White Rabbit-based architecture, its role in both data transmission and time synchronization, its implementation with the new detector components that were recently deployed and are operational since the fall of 2024, and the integration of Phase-1 and Phase-2 sectors.
Full text
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 1 The Data Acquisition System for the complete KM3NeT/ARCA neutrino telescope Francesco Benfenati Gualandi on behalf of the KM3NeT Collaboration Istituto Nazionale di Fisica Nucleare - Sezione di Bologna
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 The KM3NeT neutrino experiment 2 •ARCA (Astroparticle Research with Cosmics in the Abyss) Observation of high energy (GeV-PeV) astrophysical neutrino sources 📍Off-shore Sicily, Italy at ~3500m below sea level Determination of neutrino mass hierarchy and oscillation parameters ( O(10-100) GeV) 📍Off-shore Toulon, France at ~2500m below sea level •ORCA (Oscillation Research with Cosmics in the Abyss) KM3NeT = Km3 Neutrino Telescope ORCA ARCA > 50 Institutes > 250 physicists and engineers ARCA marine operations Currently: ARCA51 Currently: ORCA28
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 KM3NeT neutrino telescopes 3 Sea floor maps of installed DUs: • Different topologies, same detector concept Current status ARCA-51 footprint ORCA-28 footprint The UHE neutrino KM3-230213A 5/24 Full detector: 115 DUs ORCA 28 ARCA ORCA Location Italy France N. building blocks 2 1 N. DU per b.b. 115 115 DU distance 90 m 20 m DOM spacing 36 m 9 m DU height ~ 800 m ~ 200 m Instrumented mass (Mton) 2*650 7 Depth 3500 m 2500 m ARCA 51 Full detector: 230 DUs ARCA Phase-1
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 4 KM3NeT neutrino telescopes x18 Digital Optical Module x31 PMTs ~800 m (ARCA) Detection Unit Base Module “DU” “DOM” Follow I.C.Rea’s presentation tomorrow!Follow I.C.Rea’s presentation tomorrow! • Different topologies, same detector concept O(10) cm DOM positioning accuracy with acoustic positioning O(1) ns timing with “White Rabbit”
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 The Central Logic Board 5 Octopus Large: 19 PMTs and Small: 12 PMTs + piezoelectric sensor (for acoustic positioning) Capitolo 2 Sezione 2.4. PHOTOMULTIPLIER BASE PMT Figura 2.8: Grandezza di un CLB rapportato alla grandezza di una scheda di memoria. 2.4 Photomultiplier Base PMT La scheda base [7] PMT si occupa sia della generazione dell’HV che della digitalizzazione del segnale. Prima di essere digitalizzato, il segnale PMT viene amplificato da un preamplificatore integrato nella base PMT. In figura 2.12 troviamo un singolo PMT collegato alla sua base. Uno dei componenti principali della base PMT `e un comparatore, che fornisce un segnale logico alto quando l’uscita del PMT `e oltre la soglia impostata tramite I2C. La durata del segnale primario (ToT) fornito dalle basi PMT `e misurata accuratamente dai TDC del CLB. Oltre al segnale logico, la base PMT emette anche il segnale PMT analogico amplificato, che viene utilizzato solo per il test. Le 31 schede base PMT sono collegate all’SCB tramite PCB flessibile. L’HV, configurabile da remoto tramite I2C, viene generato indipendentemente in ciascuna base PMT. Ci`o consente di regolare il guadagno dei singoli PMT al fine di equalizzare la risposta dei fotoni PMT incrociati. Il valore HV pu`o essere regolato a distanza, da -800 a -1400 V [8]. La Figura 16 mostra un diagramma della scheda base PMT con i suoi componenti principali. In figura 2.11 `e mostrato un diagramma della scheda alla base del PMT con le proprie componenti. Al fine di ridurre lo spazio occupato dalla scheda base del PMT e anche il suo costo e il consumo di energia, sono stati sviluppati due Application Specific Integrated Circuits (ASIC). Il primo ASIC `e il PROMiS ASIC, con il compito di eseguire la lettura dei segnali PMT e ha due parti di↵erenti, una digitale e una analogica. Il secondo `e l’ASIC CoCo, che controlla l’alimentatore CockroftWalton HV fornendo un guadagno di 106. Nella tabella 2.1 verranno elencate le caratteristiche principali PROMiS mentre nella tabella 2.2 verranno elencate le caratteristiche dei CoCo. 29 Central Logic Board (CLB) Two LM32 cores •WhiteRabbit core for timing control • KM3NeT CLB core for DAQ control/ instrumentation readout Three DAQ modules • Time to Digital Converter (TDC) - from PMTs •AES-standard receiver - from hydrophones •MONitoring - for performance information Central Logic Board firmware architecture
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 D.A.Q. requirements 6 •Big volume •Water optical properties (absorption & scattering of blue-green photons ~50-100 m) •Good angular resolution O(0.1°) for neutrino astronomy Many optical modules O(1000/km^3) Scalable DAQ design • Trigger-less streaming readout • complex DAQ structures in harsh conditions (mandatory: minimal underwater complexity) ALL-DATA-TO-SHORE approach • signal-to-noise ratio extremely disfavoured : • muon rate (atmospheric dominating): O(100) Hz/km3 •40K decays (~constant): O(10) kHz/PMT • Bioluminescence (occasional): O(100) kHz/PMT High continuous throughput to shore: → large bandwidth switching infrastructure → strong data reduction Physical constraints Detector constraints Drawbacks
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Network general overview 7 Control Data (Commands and instrument data) DataBase (Italy and France) Local Storage Trigger and Data Acquisition System (TriDAS) Detector: DOM/Bases Off-shore On-shore (shore station facility) Control Unit (CU) Detector Data Optical, Acoustic, monitoring) CNAF (Bologna - Italy) Monitoring and On-line Multi-Messenger services Remote facilities CC-Lyon (France) Time Sync Data White Rabbit PTP (WR) GPS All detector data to shore Incoming all det. data Selected det. data “Raw” LAN Hybrid-Asym.! WR-Ethernet ! 1-10-25GbE! UDP/IP! Class A! “Control” LAN Std. Ethernet! 1-2 GbE! TCP/IP! Class C “Filtered” LAN 10-25GbE! TCP/IP! Class B! External Links 10 GbE! TCP/IP! Class C!
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 8 TriDAS general overview Aggregation ! switch switch Raw data from DOMs Step 1
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 9 TriDAS general overview Aggregation ! switch switch Raw data from DOMs Step 2
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Standard White Rabbit - topology 16 SD N SD N SD N …… Detector Control Trigger & DAQ shore station off-shore off-shore “Broadcast”-WRS DOMs Split to N DOMs “Dry”-WRS “Wet”-WRS “Broadcast” scenario (current implementation) •Not scalable to large number of strings (limited fibers in submarine cables) •Reduction of a factor 10 in the number of fibers in the submarine cable “Standard White Rabbit” scenario •Standard technology: easy debug, maintenance and updates •White Rabbit switch (WRS) specific customization required •All 1:1 connections Standard switches Detector Control Trigger & DAQ Standard switches “Broadcast” “Standard White Rabbit” shore station 1 DU SD N Base “Level-1” WRS •ARCA: up to 32 strings … … Base Module DOMS •ARCA: beyond 32 strings (currently 21 DUs)
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 17 Standard White Rabbit - technology Central Logic Board (CLB) v4.5 (Standard WR design) Glenair transceiver Custom backplane with Glenair transceivers Standard WR Switch Core Board “Wet” WRS-B Base v4 Central Logic Board “Wet” WRS-A ●Re-designed form-factor, new power boards, custom “Wet” White Rabbit switches ●23 bidirectional short range transceivers for DOM connections, CLB connection, and inter-switches connection (backup interlink)
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 OÏBNKJPAJ@AHA?PNKJE?O P=J@=N@ IQHPEH=UAN AJOKN H=UAN JODKNA Õ¿» BBODKNA ª¿»» ª¾Á»»JK@AO ½QEH@EJCHK?GO ª¾Á»»O ¡ KM3NeT White Rabbit network 18 OÏBNKJPAJ@AHA?PNKJE?O P=J@=N@ IQHPEH=UAN AJOKN H=UAN JODKNA Õ¿» BBODKNA ª¿»» ª¾Á»»JK@AO ½QEH@EJCHK?GO ª¾Á»»O ¡
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Data taking in the Standard WR architecture •ARCA Phase-2 with 3 DUs (deployed in Sept. 2024) has been stably taking data for ~8 months •ARCA Phase-2 with 21 DUs (18 new deployed DUs in summer 2025) currently under commissioning 19 Bologna test bench
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 20 Software Defined Networking • The SDN implementation is being adapted to include DUs in Standard White Rabbit. • It required to separate shore-to-sea traffic by IP address Switch “SCSF” Dom BOOTP DOM DHCP reqs DOM to CU CLB responses DOM to DQ1 DOM to DQn One rule per DataQueue Data from DOMs switch “SCBD” DOM to SCSF to Broadcast switch 2 to Broadcast switch 1 to Broadcast switch 3 CU to B-DOM shore-to-sea commands CU to SWR-DOM shore-to-sea commands DryFES Broadcast + Standard White Rabbit: merged scenario Dry WRS 1) Proper routing of the communication 2) Proper data streaming aggregation
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Conclusions •Data Acquisition in the Standard White Rabbit architecture has been on-going for almost one year in ARCA with 3 Detection Units •After the commissioning period following the recent deployment, ARCA will be running with two different architectures simultaneously 21 Thanks for your attention!
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Spare slides 22
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Detection principle 23 2.1 High-energy neutrino detection Figure 2.2: Simplified illustration of the origin of atmospheric muons and neutrinos. They are the dominant background for astrophysical neutrino detection. space-time correlation between hits in nearby photomultipliers. 2.1.1 High-energy neutrino interactions Neutrinos can interact with target nucleons through deep inelastic scattering. The di↵erent event classes produced by the di↵erent neutrino flavours are shown in fig. 2.3. The type of the interaction, charged current or neutral current, and the neutrinos flavour greatly a↵ect the signature of the neutrino event in the detector. An example of the light deposit of a typical shower-like event and a track-like event is shown in fig. 2.4. Neutral current interactions are similar for all flavours. In this case the neutrino will react with a nucleon, resulting in a lower energy scattered neutrino and a hadronic shower in the final state: ⌫l(¯⌫l)+N!⌫l(¯⌫l)+Xl=e, µ, ⌧.(2.1) Charged current interactions produce a relativistic charged lepton and a hadronic shower: ⌫l(¯⌫l)+N!l(¯ l)+Xl=e, µ, ⌧.(2.2) The emerging lepton inherits the flavour of the incident neutrino. Electron-neutrinos will therefore produce electrons and thereby an electromagnetic shower overlapping with 36 ~800 m x18 Digital Optical Module x31 PMTs x230 Detection Unit (ARCA) Base Module “BM” interaction Cherenkov light cone PMTs • Exploit Cherenkov effect in water induced by outgoing leptons • Reconstruct direction/energy from PMT hits < 20 cm positioning accuracy with acoustic pos. system < 1 ns timing with “White Rabbit” system Performances: A network of hydrophones and beacons allows, together with DOMs piezo-sensors, to determine the position of DOMs via signal triangulation “DU” “DOM”
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 D.A.Q general overview 24 Slow Control Data (Commands and instrument data) DataBase (Italy and France) Local Storage Trigger and Data Acquisition System (TriDAS) Detector: DOM/Bases Off-shore On-shore (shore station facility) Control Unit (CU) Fast Acquisition Data - FAD (Optical, Acoustic, monitoring) CNAF (Bologna - Italy) Monitoring and On-line Multi-Messenger services Remote facilities CC-Lyon (France) Time Sync Data White Rabbit PTP (WR) GPS All FAD to shore Incoming all FAD Selected FAD RAW LAN Hybrid-Asym.! WR-Ethernet ! 1-10-25GbE! UDP/IP! Class A! CTL LAN Std. Ethernet! 1-2 GbE! TCP/IP! Class C FLTR LAN 10-25GbE! TCP/IP! External Links 10 GbE! TCP/IP! White Rabbit switch sector Optical sector (mux/demux/optical amplifier) DOM Front End Switch sectors ARCA shore station Junction Box 3 rack (12 DUs) Junction Box 1 rack (8 DUs) Junction Box 2 rack (12 DUs)
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 25 KM3NeT White Rabbit network
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Test environment in Bologna 32 Broadcast "DU" Standard White Rabbit "DU"
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Test environment in Bologna 33 Rack 1 - networking Rack 2 - computing
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 34 The OctoPAES board •PMT and Acoustic Emulation of Signal • Emulates the presence of PMTs + Octopus boards (Small/Large)+piezo/hydrophone jumper for injection frequency No Octopus, PMT bases, PMTs, piezo Specific connector 10M04SCE144I7G CPLD Thanks to the dip switch it can work as LARGE or SMALL • Hit-time information encoded in a binary file • Master/Slave with clock distribution in daisy chain + start/stop in parallel • Signal (@5Hz,10Hz,1kHz,10kHz) + Background (@5kHz) • Binary file creation automatized with a GUI Track physical parameters ( ) q,θ−ϕ LVDS pulses generated by OctoPAES GUI jumper for injection frequency selection Area of dots Number of hits ∝ Injected Reconstructed Signal injected: sin wave @ 32 kHz Preliminary Height vs time Graphical User Interface
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 The Control Unit 35 The CU is a collection of (web) services which, via a state machine, drive! -the Detector! -the computing processes ! -the interactions with DB for! -runsetups, calibrations! -Instruments data logging
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 36
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 37 Frame i Frame i+1 Frame n Frame ... TS i TS i+1 ... TS n Time ΔTS: O(100 ms) DOM j • Timeslice (TS): it is the abstract subdivision of the continuity in the time-line of the experiment. • Frame: it is the group of information of a certain flavour (TDC, AES, MON) occurred in a DOM during a TS. Distributing the computational load - Each trigger algo applied to one full set of frames of one TS. - Multiple TSs handled in parallel A DQ collects data from a sector of DOMs and DU-BMs. All DQs transfer all their data from a precise Time Slice to the very same oDF. Optical World If needed more Filtering Power, just add it! Dispatcher PMT Raw data Data Writer Local storage CC-Lyon INFN CNAF Other applications (monitoring, MM) TS i TS i TS i < 20 Mbps/DOM ~3 Gbps/( DQ || ODF ) ~ 3 Mbps/( DQ || ODF ) ~ 10 Mbps x N. ODF
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 38 Frame i Frame i+1 Frame n Frame ... TS i TS i+1 ... TS n Time ΔTS: O(100 ms) DOM j • Timeslice (TS): it is the abstract subdivision of the continuity in the time-line of the experiment. • Frame: it is the group of information of a certain flavour (TDC, AES, MON) occurred in a DOM during a TS. Distributing the computational load - Each trigger algo applied to one full set of frames of one TS. - Multiple TSs handled in parallel Acoustic data must be sent in a continuous stream, addressing all data from one DQ to a single Acoustic DF. Independent reconstruction of the Time Of Arrival (TOA) of acoustic signals from various beacons Acoustic World Local storage Piezo/Hydrophone Raw data Reconstructed TOA Redundant Central DataBase DataBase Interface (Control Unit service)
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 39 32 ARCA DUs only (ARCA-Phase 1)
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 40 Optical data for Physics Acoustic data for positioning
F. Benfenati Gualandi XXI International Workshop on Neutrino Telescopes - Padova 01/10/2025 Fast Data Acquisition channels 41