EGI IRTF 2024 in Review: Incidents, Learnings, and Plans for 2025

22/01/2025 2024 in Review: Incidents, Learnings, and Plans EGI CSIRT’s IRTF EGI Incident Response Task Force Introduction Incident Response Task Force •EGI CSIRT –Coordinates operational security activities within EGI •Incident Response Task Force (IRTF) –A small team of security experts, part of the EGI CSIRT, distributed across multiple countries and organizations. –Take part in an on-duty rota, act as first responders to reports of security incidents within the EGI Infrastructure. –Incident response and digital forensics expertise is made available to sites for the investigation and resolution of incidents. 3 https://csirt.egi.eu/activities/ Vulnerabilities •Pakiti agents are deployed on computing nodes to monitor and report the patching status of Linux systems. •30 Advisories sent by SVG in 2024 •53 Vulnerabilities reported by IRTF in 2024 –7 Critical,! 17 High,! 1 Moderate,! 28 Others (unspecified, advise on config change). 4 https://pakiti.egi.eu/ https://operations-portal.egi.eu/ Communications Challenge •Ensure that contact information is up-todate and functional (biannual activity) –Enabling efficient coordination during an incident. •It is an email containing details about the challenge along with a unique URL. –Recipients are required to access the URL, which allows the response time to be recorded. 5 Communications Challenge •New Procedure: •EGI-Operations is coordinating the follow-up. Many thanks! 6 Communications Challenge - Sites •The EGI Incident Response policy states: –“You shall follow the incident response procedure defined by the e-Infrastructure". •The associated procedure (SEC01 EGI CSIRT Security Incident Handling Procedure) defines a maximum response time of 4 hours. •EGI sites not responding promptly to security notifications are being suspended in GOC-DB •Only Certified sites are being tested. –+30% of the sites are not tested! Similar for Pakiti –This information is not consumed by VOs! 7 https://documents.egi.eu/public/ShowDocument?docid=2935 https://confluence.egi.eu/display/EGIPP/SEC01+EGI+CSIRT+Security+Incident+Handling+Procedure https://goc.egi.eu Communications Challenge - VOs •Community Operations Security Policy covers the need to define a security contact and reply to security-related requests in a timely manner. –There are no direct penalties for not replying in due time. –We rely on building closer relationships with VOs to demonstrate the importance of responding to such tests and keeping their contact information up to date. 8 https://confluence.egi.eu/display/EGIPP/Community+Operations+Security+Policy Results Communications Challenge - Sites •Excluding technical errors, 13 sites (6%) and 30 sites (13%) failed to respond. •9 (4%) did not respond to either of the two campaigns. • 9 16 Incident #1 Identity Mismanagement EGI Incident Response Task Force [email protected] https://confluence.egi.eu/display/EGIBG/CSIRT+PGP+key Identity Mismanagement •All users from a university IdP were assigned the same EGI Check-in account –This was due to the common definition of voPersonID. –To mitigate this issue: •The voPersonID field was initially disabled for users coming from this IdP. •User identification was switched to rely only on eduPersonUniqueId for that specific IdP. –The logs of services accessed using the shared Check-in identity were analysed and no malicious activity was detected. –After the university implemented the solution, the incident was considered as resolved. 17 18 IncidentS #2 Users Misbehaving EGI Incident Response Task Force [email protected] https://confluence.egi.eu/display/EGIBG/CSIRT+PGP+key EGI Check-in users misbehaving •A user attempts to access multiple, unconnected VOs’ resources. A. Compromised account B. Generic account C. Legit account •The IdP was contacted, suspended the user and deleted their active sessions. •There was no sign of any further abuse. •Some VOs which granted access to the user were unresponsive (also on the CommsChallenge) D. The VOs provided access to resources on a "free trial" basis. E. The user gained access with a credible narrative and a legitimate passport. 19 20 Incident #3 Site Compromise EGI Incident Response Task Force [email protected] https://confluence.egi.eu/display/EGIBG/CSIRT+PGP+key Detection •Unusual high load triggered suspicion •Admin reported and asked for help •Access to a forensics proxy was provided 21 IRTF Suspicious Server Forensics Proxy Initial Checks •No exceptional load •No suspicious processes •No unusual ports opened •…but one connection without PID 22 Rootkit detection 23 Payload 24 kernel-dbus_start.sh Hide process Rootkit injection Deactivate SELinux SSH stealer C2 connection Miner trigger Wipe logs Check for connections and stop the miner Impact •Most of the nodes of the whole infrastructure were compromised 25 Execution and Persistence •Manual trigger •Scheduled Task/Job •Boot or Logon Autostart Execution •Kernel Modules and Extensions –https://github.com/m0nad/Diamorphine •Add SSH Authorized Keys 32 Defense Evasion •Masquerading file name, service and location •Hide Artifacts •Obfuscated Information –Open source: node-bash-obfuscate 33 https://github.com/willshiao/node-bash-obfuscate Defense Evasion •Disable firewall •Competition removal •Clean Logs and delete malware –Open Source: mig •Use non-Standard Port 34 https://github.com/Kabot/mig-logcleaner-resurrected Privilege escalation •Valid Accounts –Compromised accounts with admin/sudo privileges. •Exploitation for Privilege Escalation –OpenSource vulnerability explorer. 35 https://github.com/The-Z-Labs/linux-exploit-suggester/linux-exploit-suggester.sh Command and Control •Bidirectional Communication using IRC (Internet Relay Chat) –IRC is a protocol using TCP/IP for real-time text-based communication 36 IRC Connect Infected Server Victim infrastructure IRC Legit network https://undernet.org/ Payload •Network Flood (DDoS) –MrScytheLULZ •Crypto mining –Nanominer and GMiner 37 One month during the incident https://github.com/nanopool/nanominer https://github.com/develsoftware/GMinerRelease https://github.com/MrScytheLULZ/DDoS-Scripts Attack Recipe 38 Attack Phase Open Source Tool Initial Access SSHPrank, CBruteKrag Rootkit Diamorphine Defense Evasion Node-Bash-Obfuscate, Mig-Logcleaner Lateral Movement Masscan Privilege escalation TheZLabs Exploit Suggester Crypto mining Nanominer, GMiner DDOS Pearlbot IRC Undernet, Dalnet Impact •Operating since at least 2020 •200+ Linux servers compromised •25+ organizations impacted •Two groups operating the same tools •Attack spread quickly •Found $16 000 in a single crypto wallet 39 Research & Education Threat Intelligence Sharing •Advisories –Share detailed information –Detection and mitigation instructions •MISP –Share IOCs for immediate detection •WLCG Security Newsletter –Coming soon. Subscribe! 40 https://misp.cern.ch https://e-groups.cern.ch/e-groups/EgroupsSubscription.do?egroupName=wlcg-security-newsletter Learnings Initiatives •Develop a strategy with EGI Operations and VOs to improve site logging capabilities. •Conduct lightweight security exercises to test and strengthen site incident response capabilities. •Collaborate with WLCG to strengthen relationships with VOs. –Discuss security topics in the WLCG Open Technical Forum (OTF). –Publish the WLCG Cybersecurity Newsletter regularly. Subscribe! –Review policies and procedures to ensure alignment with evolving needs. •Organize trainings including conferences, tCSC, and hands-on workshops. •Implement automated scans to monitor EGI central services proactively. 48 https://e-groups.cern.ch/e-groups/EgroupsSubscription.do?egroupName=wlcg-security-newsletter Hands-On Workshop •Technical workshop for sysadmins •Learn the most important points to –Respond effectively to an incident –Gather evidence without tampering it –Handle and structure investigations data –Perform digital forensics analysis 49 https://indico.cern.ch/e/security-workshop25 https://e-groups.cern.ch/e-groups/EgroupsSubscription.do?egroupName=security-workshop25 50 Questions / Feedback EGI Incident Response Task Force [email protected] https://confluence.egi.eu/display/EGIBG/CSIRT+PGP+key