Barrel muon track reconstruction at CMS level-1 trigger using deep learning

BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING September 2024 AUTHOR(S): Fabio Cufino TU Dortmund University CMS Collaboration SUPERVISOR(S): Rocco Ardino Thomas Owen James CERN openlab Report x/2024 Abstract The CMS Level-1 (L1) Trigger executes rapid (under 3 microseconds) reconstruction using FPGA logic on a subset of detector data at the 40 MHz bunch-crossing rate, selecting approximately 100 kHz of the most interesting events for full readout and processing in the software filter farm. The L1 Scouting system introduces a novel data-taking paradigm, capturing L1 information at the full 40 MHz rate, enabling analyses with minimal selection bias and greater flexibility for studying rare or unexpected phenomena. This project uses machine learning techniques to implement a robust real-time data quality and system integrity monitoring solution, ensuring precise oversight of FPGA operations within L1 Scouting. The integration of these methods supports efficient and reliable operation of the system, enabling detailed physics analyses under high-luminosity conditions. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 1 CERN openlab Report x/2024 Table of contents 1 Introduction 3 2 The CMS experiment at the Large Hadron Collider 3 3 CMS Level-1 Trigger and Data Scouting 5 3.1 L1Trigger....................................... 5 3.2 DataScouting..................................... 6 3.2.1 The CMS Run-3 Level-1 Data Scouting Demonstrator . . . . . . . . . . . 7 4 BMTF Track Reconstruction 10 4.1 KalmanFilter..................................... 10 5 Neural Networks for Muon Reconstruction 12 5.1 Dataset ........................................ 12 6 Neural Network Design and Implementation 13 6.1 Results......................................... 14 6.2 Conclusion....................................... 16 BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 2 CERN openlab Report x/2024 1 Introduction The Compact Muon Solenoid (CMS) experiment is one of the primary multi-purpose detectors at the Large Hadron Collider (LHC). The LHC, colliding proton bunches at a nominal frequency of 40 MHz, generates an immense volume of collision data, necessitating an efficient system to process and filter this information. To achieve this, CMS employs a dual-tiered trigger system designed for event selection. The initial tier, known as the Level-1 trigger (L1T), is hardware-based and utilizes algorithms with fixed latency to perform basic object reconstruction, primarily focusing on data from calorimeters and the muon system to discard events that are unlikely to be of physical interest. After a successful L1T selection, the full detector is read out, and the raw data is sent to the subsequent tier, the High-Level Trigger (HLT). HLT uses software to conduct the online reconstruction, leveraging the full array of detector information to select events that meet specific criteria [5]. Given the sheer volume of data produced by the detector, the CMS trigger system must implement must implement selection criteria to target a generic set of interesting signature. While this approach increases the probability of capturing known physics processes, it runs the risk of excluding potentially rare phenomena. To address this, data scouting has been introduced as a method for extracting less-filtered event information directly from the trigger system, offering a broader scope for analysis. As the LHC prepares for its High-Luminosity (HL-LHC) phase, expected to begin later this decade, substantial upgrades are being implemented across the CMS experiment, particularly within the trigger system. These upgrades aim to maintain and extend the standard CMS physics program under larger pileup conditions at the price of an increase in the L1 accept rate from approximately 100 kHz to 750 kHz. The Level-1 trigger will undergo significant changes, incorporating advanced algorithms and reading out particle tracks from the tracker back-end, with the goal of achieving near-offline reconstruction quality at 40 MHz [2], [13]. This development makes Level-1 trigger Data Scouting (L1DS) particularly relevant. L1DS is poised to play a crucial role in future physics analyses by providing data that is nearly unbiased, providing a promising ground for real-time analysis and anomaly detection applications. In preparation for these HL-LHC operations, the CMS collaboration has developed a Run-3 demonstrator for the L1DS system. This platform is designed to be compatible with the current trigger system architecture and serves as a testing ground for scouting strategies, hardware configurations, and data extraction techniques in anticipation of the upcoming upgrades. The report explores the application of machine learning techniques to reconstruct muon tracks from segments based on super-primitives (or stubs) which are processed by the TwinMux and delivered to the Barrel Muon Track Finder (BMTF), evaluating their potential for integration into the enhanced L1DS system planned for future high-luminosity operations. 2 The CMS experiment at the Large Hadron Collider The Large Hadron Collider (LHC), situated at CERN in Geneva, Switzerland, resides within a 27 Km tunnel buried 100 meters underground. Operational since 2010, the LHC is capable of accelerating protons and heavy-ion beams to reach a center-of-mass energy up to 13.6 TeV [4]. The procedure of injecting protons into the LHC is facilitated by a sequence of accelerators, which enable an injection energy of 450 GeV into the collider. The typical beam structure comprises 39 trains, each containing 72 bunches with N= 1.1×1011 protons. Operating at a crossing frequency of 40 MHz, the setup results in an inter-collision interval of 25 ns. At the LHC’s nominal instantaneous luminosity, Linst = 2 ×1034 cm−2s−1, multiple proton-proton BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 3 CERN openlab Report x/2024 interactions occur in a single bunch crossing, a phenomenon referred to as ’pile-up.’ Under these conditions, the LHC achieves an average pile-up of approximately 60 interactions per bunch crossing. Collisions are managed at four specific interaction points within the accelerator ring, where the main experiments of the LHC—ATLAS, CMS, ALICE, and LHCb—are situated. The CMS - Compact Muon Solenoid - is one of the four pivotal detectors operational at the LHC. It serves as a versatile detector, designed with multiple concentric layers, each aimed at capturing and analyzing distinct properties of the particles generated from highenergy collisions. The CMS detector is equipped with various subsystems: •ASilicon Tracker, sensitive to charged particles, enabling the reconstruction of their trajectories. •An Electromagnetic Calorimeter (ECAL), which measures the energy of electrons and photons. •AHadronic Calorimeter (HCAL), which measures the energy of hadrons. •ASuperconducting Solenoid, producing a 3.8 T magnetic field, which bends the paths of charged particles in the transverse plane, allowing their momentum to be measured. •AMuon System, designed to capture muon tracks, consisting of Drift Tubes (DTs), Cathode Strip Chambers (CSCs), Resistive Plate Chambers (RPCs) and Geseous Electron Multipliers (GEMs). Figure 1showcases the diverse subdetectors mentioned and other components [15]. Figure 1: Schematic rapresentation of the CMS detector [9] BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 4 CERN openlab Report x/2024 3 CMS Level-1 Trigger and Data Scouting The CMS Trigger System is pivotal in the data acquisition process, as it reduces the vast amount of collision data generated at a frequency of 40 MHz to a manageable offline storage rate of approximately 1 kHz [10]. This reduction is essential due to the limitations in readout and storage capabilities. The trigger system consists of two main components: the Level-1 Trigger and the High-Level Trigger. The L1T, built with custom electronics, prioritizes swift decision-making by utilizing coarse-grained muon detectors and calorimeter data to decrease the event readout rate to 100 kHz. In contrast, the HLT is software-driven and operates on a dedicated processor farm. It leverages detailed information from all subdetectors, including the silicon inner tracker, to enable a more thorough analysis of events. 3.1 L1 Trigger The L1T [8] system is designed to evaluate each bunch crossing and determine whether events should be retained for further analysis. It consists of local, regional, and global triggers, each contributing to a structured decision-making process. Local Triggers, or Trigger Primitive Generators (TPGs), identify candidates by analyzing energy deposits in calorimeter towers and hit patterns in muon chambers. Regional Triggers aggregate this information, applying spatial pattern logic to assess and rank the trigger objects, such as muon candidates, based on parameters like energy and momentum. The final decision is made by the Global Trigger (GT), which employs algorithms to determine if an event meets the criteria for further analysis, producing a Level-1 Accept (L1A) signal if the conditions are met. This process is performed within a 4-microsecond window, relying on custom hardware, including FPGAs and LUTs, to ensure timely operation. Within this framework, the Muon and Calorimeter Triggers perform specific roles. Figure 2showcases a schematic representation of the Muon and Calorimeter Triggers. Muon Trigger Calorimeter Trigger Figure 2: Diagram of the upgraded CMS Level-1 trigger system during Run 2 [12] BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 5 CERN openlab Report x/2024 The Muon Trigger system is designed to accurately track muons across different regions of the detector. It is divided into three subsystems, each targeting distinct ηranges: the Barrel Muon Track Finder (BMTF), the Overlap Muon Track Finder (OMTF), and the Endcap Muon Track Finder (EMTF). These subsystems process Trigger Primitives (TPs) generated from the CSCs, DTs, and RPCs. CSCs provide TPs to the EMTF and OMTF via a mezzanine on the muon port card, while RPC hits from the endcaps are processed through the Concentrator Pre-Processor and Fan-out (CPPF) card. Barrel RPC and DT TPs are routed to the TwinMux concentrator card, which combines the precise spatial resolution of DT segments with the timing characteristics of RPC hits to produce super-primitives, that are track segments made from combining hits in both the drift tube and resistive plate chamber sections of the detector. These refined data are then utilized by the BMTF and OMTF to enhance muon candidate identification. The system maintains redundancy by using multiple detection chambers, with additional stations extending coverage up to |η| = 2.8. The final step involves the Global Muon Trigger (GMT) that eliminates duplicates, arranges muons, selects the top eight muon candidates, which are then forwarded to the Global Trigger (GT) for further analysis. 3.2 Data Scouting The L1 Data Scouting system introduces a novel approach to data collection by capturing intermediate data directly from the L1 trigger at the full 40 MHz bunch crossing rate. This allows for the analysis of high-frequency event types that are typically too common to be included in the standard L1 trigger menu. Notably, L1DS operates independently of the L1 trigger decision-making process, meaning it does not influence the outcomes of the trigger. Figure 3: CMS Level-1 (L1) scouting scheme [6] With L1DS, it’s possible to perform semi-real-time analysis using L1 trigger objects or store compact event records. Additionally, the system enhances diagnostic and monitoring capabilities, providing BX-to-BX correlations and enabling independent per-bunch luminosity measurements. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 6 CERN openlab Report x/2024 3.2.1 The CMS Run-3 Level-1 Data Scouting Demonstrator A L1DS demonstrator has been set up for Run-3. This demonstrator enables event analysis directly at the L1T stage by capturing data at the full 40 MHz bunch crossing rate, independent of the primary L1 trigger decision-making process. Figure 4shows the scheme for the L1DS demonstrator. CPPF TwinMux Layer 1 Calorimeter Trigger BMTF OMTF EMTF Layer 2 Calorimeter Trigger GMT DeMux GT BMTF x12 x6 GT x7x18 x24 x4 BRIL-DS DS-Ctrl USC55 SCX55 VCU128-1 VCU128-2 SB852 PP Run-3 DS15 April config 10 Gb/s trigger link protocol 100 Gb/s Ethernet (CWDM4) 100 Gb/s Ethernet 100 Gb/s RoCE DSBU DSBU DSBU DSBU DSPU DSPU DSPU DSPU DSPU DSPU Cluster FS (Lustre) Figure 4: The CMS Run-3 Level-1 Data Scouting demonstrator architecture. The architecture of the L1DS demonstrator integrates several layers and components designed to optimize the capture and processing of data streams from the L1T system. The system is set up to draw data from at various stages of the L1 trigger, specifically from the Global Trigger, Global Muon Trigger, Calorimeter, and Barrel Muon Track Finder, with details illustrated in Table 1. To manage the drawn data efficiently, the L1DS system is built on a foundation of highbandwidth data paths. These paths employ 10 Gb/s trigger link protocols to rapidly transmit raw data from the trigger systems. The system utilizes 100 Gb/s Ethernet connections for the output link between the FPGA boards and the Data Scouting Buffer Units (DSBUs) located on the surface, ensuring the swift transfer of data necessary to handle the enormous volumes generated.. At the heart of the data processing within the L1DS system are Xilinx VCU128 FPGA boards, which feature the powerful Ultrascale+ XCVU37P FPGA. These boards are tasked with processing the data at high speeds, leveraging their GTY transceivers and multiple QSFP ports for high-bandwidth connectivity. The VCU128 boards also include 8 GB of High Bandwidth Memory (HBM), which serves as a large temporary buffer for the data, later transmitted via TCP/IP through 100 GbE links to the Data Scouting Buffer Units (DSBUs) for further processing [11]. As the data moves through the system, after the FPGA boards, it first reaches the Data Scouting Buffer Units (DSBUs). These units are designed to temporarily store the data streams coming from the scouting boards, performing basic processing and preparing the data fragments to be injected into the Data Scouting Processing Units for more intensive processing. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 7 CERN openlab Report x/2024 Input system N 10Gb/s links Objects GMT 4 Up to 8 GMT muons Calorimeter trigger 7 + 1 spare e/γ, tau candidates, jets and energy sums including Emiss T BMTF 24 BMTF input super primitives GT 18 Algorithm bits Table 1: Inputs to the L1DS demonstrator. Following this initial buffering, the data is processed by the DSPUs. These units unpack the raw data fragments produced by the DSBUs and aggregate them into collections designed for efficient storage. Additionally, they perform online analysis on the aggregated data to identify sets of BXs that meet specific analysis requirements. These analysis processes produce a set of BXs that meet the criteria, and the filtered BXs are stored in an additional data stream. The main data stream, called the ZeroBias stream, contains unpacked data that has not been filtered or analyzed and is prescaled to limit total output throughput. Finally, the processed data is temporarily stored in the Lustre Cluster File System, which is used as intermediate storage before being moved to the Tier-0 for repacking and permanent storage in the CMS Data Aggregation System. TwinMux System in the CMS Level-1 Muon Trigger Upgrade The TwinMux system is a key component of the upgraded Level-1 (L1) muon trigger system for the CMS experiment, particularly in the barrel region of the detector. Serving as a “barrel concentrator”, the TwinMux boards aggregate and process data from DTs and RPCs muon detection subsystems. The system consists of 60 TwinMux boards strategically deployed to handle inputs from these different subdetectors and ensure efficient data processing and transmission to the BMTF. Each TwinMux board is equipped to handle inputs from both DTs and RPCs, converting and concentrating the lower-speed optical trigger links into higher-speed links that are sent to the BMTF and Overlap Muon Track Finder (OMTF). This conversion is crucial for managing the high data rates required by the upgraded CMS trigger system. By concentrating data into fewer, faster links, TwinMux helps to streamline the transmission process, reducing the number of physical connections needed and increasing the system’s reliability and performance under high-luminosity conditions [16]. A picture of a TwinMux board is showed in Figure 5. Figure 5: TwinMux board picture BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 8 CERN openlab Report x/2024 (a) (b) (c) Figure 7: Transverse momentum reconstruction resolution for the machine learning method (NN reconstruction) and the traditional Kalman Filter approach (L1) used by the BMTF. The plots show the comparison for neural networks using (a) 2 stubs, (b) 3 stubs, and (c) 4 stubs. (a) (b) (c) Figure 8: Residuals of pseudorapidity (η) for the neural network (NN) reconstruction and the traditional Kalman Filter approach (L1) for different stub configurations: (a) 2 stubs, (b) 3 stubs, and (c) 4 stubs. (a) (b) (c) Figure 9: Residuals of azimuthal angle (ϕ) for the neural network (NN) reconstruction and the traditional Kalman Filter approach (L1) for different stub configurations: (a) 2 stubs, (b) 3 stubs, and (c) 4 stubs. The NN reconstruction achieves better performance compared to the conventional method, particularly at higher stub counts. those produced by the conventional approach. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 15 CERN openlab Report x/2024 Specifically, the neural network predictions result in narrower and more centralized distributions, indicating higher accuracy and reduced systematic errors. This improvement is particularly evident in the ϕresolution, where the traditional method exhibits a noticeable splitting in the residuals, which are effectively mitigated by the neural network’s reconstruction capabilities. To further assess the performance of the neural network in transverse momentum reconstruction, the pTresolution was examined as a function of the reconstructed pRECO T, as shown in Figure 10. (a) (b) (c) Figure 10: Comparison of the relative transverse momentum resolution, ∆pT/pT, between the L1 trigger system (L1) and the neural network (NN) reconstruction as a function of the reconstructed muon transverse momentum, pRECO T. The plots illustrate the performance for different stub configurations: (a) 2 stubs, (b) 3 stubs, and (c) 4 stubs. The results demonstrate that the neural network achieves a more consistent and precise pTreconstruction across the entire pTspectrum, with narrower uncertainty bands and lower fluctuations, particularly at lower pTvalues. Interestingly, the 3NN and 4NN configurations perform well but not as well as the 2NN, which is due to the increasing precision of the Kalman Filter used in the traditional reconstruction method as the number of stubs increases. This plot compares the relative pTresolution, ∆pT/pT, for the L1 trigger system and the neural network. The results demonstrate that the neural network achieves a more consistent and precise pTreconstruction across the entire pTspectrum. The shaded regions in the plot represent the uncertainty bands for each method, where the neural network exhibits significantly narrower bands, indicating lower uncertainty and higher reliability. Conversely, the L1 trigger system shows larger fluctuations, particularly at lower pTvalues. This consistency in performance underscores the robustness of the neural network approach in maintaining high precision across different pTranges. Interestingly, the 3NN and 4NN configurations perform well, but not as effectively as the 2NN configuration. This is likely because, as the number of stubs increases, the Kalman Filter used in the traditional reconstruction method becomes more precise, making further gains from the neural network less significant. 6.2 Conclusion The integration of neural networks into the CMS Level-1 trigger system represents a significant advancement in the evolution of data acquisition and processing techniques in high-energy physics experiments. By leveraging the power of machine learning and modern hardware accelerators, the CMS experiment is well-positioned to meet the challenges of the HL-LHC era, potentially setting new standards for real-time data processing in particle physics. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 16 CERN openlab Report x/2024 REFERENCES References [1] Rocco Ardino, Thomas James, and Nicolò Lai. Barrel muon track reconstruction with deep learning for Level-1 trigger data scouting in the CMS experiment. CERN Summer School Report. Geneva, Switzerland: CERN, 2023. [2] Rocco Ardino et al. “CMS Level-1 trigger Data Scouting firmware prototyping for LHC Run-3 and CMS Phase-2”. In: Journal of Instrumentation 19.05 (May 2024), p. C05027. doi:10.1088/1748-0221/19/05/C05027.url:https://dx.doi.org/10.1088/17480221/19/05/C05027. [3] Michalis Bachtis. “A Kalman filter for the CMS Muon Trigger for Run III and HL-LHC”. In: Research Techniques Seminar. Presented at FNAL on October 23, 2018. University of California, Los Angeles. Oct. 2018. url:%5Curl%7Bhttps://indico.fnal.gov/event/ 18738/contributions/48810/attachments/30466/37474/KalmanFilter_RTS.pdf% 7D. [4] CERN: The place for scientific curiosity. Brochure CERN-Brochure-2021-004-Eng. https: //cds.cern.ch/record/2809109/files/CERN-Brochure-2021-004-Eng.pdf. CERN, 2021. [5] CMS Collaboration. “Development of the CMS detector for the CERN LHC Run 3”. In: sub. to JINST (2023). arXiv: 2309.05466.url:https://cds.cern.ch/record/ 2870088. [6] CMS Collaboration. “Performance of the CMS Level-1 Trigger”. In: (2024). url:https: //cds.cern.ch/record/2916191/files/DP2024_096.pdf. [7] CMS Collaboration. Performance of the CMS muon detector and muon reconstruction with proton-proton collisions at √s= 13 TeV at the LHC. Tech. rep. CMS-DP-2016/074. Accessed: 2024-08-28. CERN, 2016. url:https://cds.cern.ch/record/2239285/ files/DP2016_074.pdf. [8] CMS Collaboration. “The Phase-2 Upgrade of the CMS Level-1 Trigger”. In: (2017). arXiv: 1609 . 02366 [physics.ins-det].url:https : / / inspirehep . net / files / f96f925080108949b5cc608ebcf43546. [9] CMS Open Data Workshop. Introduction to Physics Objects. Accessed: 2024-08-28. 2023. url:https://cms-opendata-workshop.github.io/workshop2023-lesson-physicsobjects/01-intro-poet/index.html. [10] The CMS Collaboration. “The CMS Run-3 Level-1 Data Scouting Demonstrator”. In: Journal of Instrumentation 19.05 (2024), p. C05027. doi:10.1088/1748-0221/19/05/ C05027. [11] Xilinx Inc. VCU128 Evaluation Kit.https://www.xilinx.com/products/boards-andkits/vcu128.html. Accessed: 2024-08-21. 2023. [12] Hyejin Kwon. “Performance of the CMS Level-1 Trigger during Run 2”. In: Feb. 2021, p. 806. doi:10.22323/1.390.0806. [13] Dinyar Sebastian Rabady et al. A 40 MHz Level-1 trigger scouting system for the CMS Phase-2 upgrade. Geneva: CERN, 2023. url:https://cds.cern.ch/record/2816252. [14] FAST Machine Learning for Science. HLS4ML: Machine Learning Inference in FPGAs. https://fastmachinelearning.org/hls4ml/. Accessed: 2024-08-27. 2024. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 17 CERN openlab Report x/2024 [15] The CMS Detector.https://cms.cern/detector. Accessed: 2024-08-23. 2024. [16] A. Triossi et al. “The CMS Barrel Muon trigger upgrade”. In: Journal of Instrumentation 12.01 (2017). Published by IOP Publishing for Sissa Medialab, p. C01095. doi:10.1088/ 17480221/12/01/C01095.url:https://doi.org/10.1088/17480221/12/01/ C01095. BARREL MUON TRACK RECONSTRUCTION AT CMS LEVEL-1 TRIGGER USING DEEP LEARNING 18