Technical Performance Metrics of a Security Operations Center

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Technical Performance Metrics of a Security Operations Center © 2023 The Author(s). Published by Elsevier Ltd. Published version Forsberg, Joonas; Frantti, Tapio Forsberg, J., & Frantti, T. (2023). Technical Performance Metrics of a Security Operations Center. Computers and Security, 135, Article 103529. https://doi.org/10.1016/j.cose.2023.103529 2023 Computers & Security 135 (2023) 103529 Available online 6 October 2023 0167-4048/© 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Contents lists available at ScienceDirect Computers & Security journal homepage: www.elsevier.com/locate/cose Technical performance metrics of a security operations center Joonas Forsberg, Tapio Frantti ∗ University of Jyväskylä, Seminaarinkatu 15, FI-40014, Jyväskylä, Finland A R T I C L E I N F O A B S T R A C T Keywords: Security operations center Soc Csoc Cyber security operations center Metric Measurement Technical performance Cyber defense Performance indicator This research introduces a novel framework for creating metrics intended for security operations centers (SOCs). The framework is developed using the design science research methodology and has been validated by generating four novel metrics to assess the technical performance of a SOC. Additionally, the study examines the existing landscape of metrics for SOCs and concludes that a majority of the metrics discussed in the literature primarily focus on operational aspects rather than technical performance. The absence of adequate technical performance metrics makes it challenging to accurately evaluate the tangible impact of a SOC on overall cyber defense capabilities. The research also highlights the insufficiency of current methods in constructing metrics and frameworks tailored for measuring SOCs’ technical performance. The resulting framework offers SOCs means to create high-quality metrics for performance evaluation. Furthermore, the metrics the framework was validated with offer SOCs an opportunity to enhance their ability to quantify their threat detection capabilities. 1. Introduction Cyber threats have evolved dramatically over the last few years. They have become more sophisticated and complex, and as a result, they have a greater impact on operational activities. The use of generalpurpose malware has declined recently and more advanced threats, such as supply chain compromises, extortion activities, misinformation campaigns, and business e-mail compromises, are continuously increasing (European Union Agency for Cybersecurity, 2021). The increased sophistication and the associated risks require organizations to improve their defensive capabilities for combatting the evolving threat landscape because traditional malware and network defenses are not enough to protect the organization from modern-day cyber threats. A strategy for organizations to increase their cyber defense capabilities is to acquire security operations center (SOC) capabilities. On a general level, the SOC operations can be produced in-house, outsourced to a managed service provider, or operated in a hybrid model in which the internal SOC is complemented by an external SOC provider. According to Nathans (2014), a SOC is typically responsible for detecting security incidents and initiating related incident response activities. Nathans also determined that depending on the size and the needs of the organization, the SOC can consist of a single person or a larger team working in 24/7 shifts that together form a coherent collection of different competencies to prevent, detect, and resolve cyber threats. Furthermore, Vielberth et al. (2020) mentioned there are several ways * Corresponding author. E-mail addresses: joonas.forsberg@iki.fi (J. Forsberg), tapio.k.frantti@jyu.fi (T. Frantti). to operate and design SOCs, and that the outcome is influenced by various factors, such as regulations, company strategy, and expertise. To quantify the operational efficiency and capabilities of a SOC, performance should be measured using a set of commonly agreed metrics. A literature survey performed by Vielberth et al. (2020)concluded that the currently established metrics are insufficient for measuring the performance of the SOC. This viewpoint is supported by Kokulu et al. (2019), Agyepong et al. (2020)and Sundaramurthy et al. (2015). Multiple publications on SOCs focus primarily on quantitative volume and time-based operational and vulnerability-related metrics to measure the performance of SOCs (Nathans, 2014; Ahlm, 2021; Kokulu et al., 2019). The operational metrics can help managers measure and optimize human resource utilization in analysis. Other quantitative metrics, such as vulnerability-related data, can provide an overview of the overall exposure to known threats (Nathans, 2014). In practice, they are inefficient in measuring the capabilities of SOCs, because they cannot measure the effectiveness of the detection capabilities and other protective controls. Measuring the false-positive rate of the monitoring rules is a common solution to combat this issue (Nathans, 2014; Ahlm, 2021). However, considering the false-positive rate alone can be misleading. The outcome of such metrics can be manipulated either subconsciously or consciously by the people responsible for the creation of the detection capabilities. Several studies have been performed on general security metrics (Böhme, 2010; Pendleton et al., 2016; Salmi, 2018), and there are also commonly referenced industry standards, such https://doi.org/10.1016/j.cose.2023.103529 Received 2 June 2023; Received in revised form 28 August 2023; Accepted 3 October 2023 Computers & Security 135 (2023) 103529 2 J. Forsberg and T. Frantti as ISO/IEC 27004:2016 (2016)and NIST SP 800-55 (Chew et al., 2008), that organizations can use to measure the effectiveness of their information security. An argument could be made that an industry standard framework for measuring SOCs does not exist at the moment. The lack of commonly accepted methods to measure the technical performance of a SOC is particularly prominent when the SOC is being outsourced to a third-party vendor. This often leads to a situation where the tendering process produces a suboptimal result, either by selecting the vendor with the lowest total service cost or the most convincing sales material. In Finland, a trend has been observed, where vendors are chosen on a proof of concept phase, in which an attack simulation is performed in an environment temporarily monitored by the vendor and afterward their threat detection capabilities are evaluated. The trend is essentially a manifestation of the difficulty in evaluating the technical capabilities of different vendors. Although it can provide useful insights when comparing vendors, the results are not truly comparable between vendors because the number of resources allocated to the proof of concept can significantly impact the final results and thus skew the vendor selection. The objective of this study was to determine the metrics and other performance indicators relevant for measuring the technical performance of a SOC. Without properly constructed metrics, it is difficult to evaluate how substantial impact the SOC has on the overall cyber defense capabilities. A technical performance metric within this paper is defined as a qualitative or quantitative indicator derived from one or more measurements resulting from the activities performed by the SOC, which describes how well the SOC can utilize technologies to prevent, identify, detect, and respond to cyber threats affecting the organization. Compared to an operational performance metric, a technical performance metric attempts to quantify the defensive capabilities, rather than assessing the effectiveness of processes and people associated with the SOC. Understanding both the operational and technical efficiency of the SOC is required to successfully protect an organization from cyber threats and measure the effectiveness of the SOC. The objective leads to the following research questions: (1) What frameworks are available to measure the performance of a SOC? (2) What are the commonly mentioned key metrics used to measure a SOC? (3) Can the common metrics be used to measure the technical performance? (4) How can the metrics be improved to enhance the reporting capabilities of technical performance? The expected outcome of the research is a novel framework that could contribute to an industry-standard way to measure the technical performance of a SOC and apply to most SOCs. 2. Research methodology Design science research methodology is used as the primary research methodology in this research. The research follows the principles defined by Peffers et al. (2007), in which a design science artifact is generated as an outcome of an iterative process with the following activities: (1) Identify the problem and motivation, (2) Define the objectives of a solution, (3) Design and development, (4) Demonstration, (5) Evaluation and (6) Communication. The first activity is covered in sections 1, 3and 4of this research, the second in section 5, the third in section 6, the fourth in section 7and the fifth in sections 8and 9. This research as a whole covers the sixth activity. The second activity is supported by a brief literature review of both academic publications and commercial sources to establish sufficient theoretical background on the subject, and to answer the research questions (1) and (2). A keyword search is used as the search method for the discovery of literature related to SOCs and metrics. The search query for SOC-related literature is ‘(security OR cyber security) (operations OR operation) (center OR centre)’ and the metric-related search query is ‘(metric OR measurement OR “performance indicator”)’. The literature review was performed between May and September in the year 2022 and updated in August 2023. JYKDOK,1IEEE Xplore2and Google Scholar3were used as the search engines. The research is expected to result in a design science artifact, a framework that can be utilized to create metrics to measure the technical performance of SOCs. The design science artifact is validated following the guidelines defined by Hevner et al. (2004), by following the experimental evaluation method, which consists of controlled experiments and simulations. The artifact is evaluated by utilizing it to create metrics to measure the technical performance of a SOC. The resulting metrics are evaluated with simulated data to ensure they behave as expected and produce meaningful insights for SOCs. The metrics are further evaluated in a controlled environment to ensure the measurements required for the metrics can be collected in a fully operational SOC. 3. Security operations center To tackle the first phase of the design science research, problem identification, we first introduce, how technical performance metrics target environments, i.e., SOCs, are described in the literature. One method to describe a SOC is through a People, Processes, and Technologies (PPT) framework (Knerler et al., 2022; Vielberth et al., 2020). Vielberth et al. (2020) summarized that the people block describes the people associated with the SOC and their required competencies, the process block describes how the people interact and how security incidents are handled, and the technology block describes the tools used to do the work. They also argued that the PPT framework could be expanded to include governance and compliance, enabling organizations to utilize the SOC as a function to ensure compliance with various standards, such as ISO/IEC 27001, GDPR, or PCI-DSS. Vielberth et al. (2020) stated, that the work performed at a SOC is heavily driven by processes, as the work is structured around the prevention, detection, and response of security incidents. The Computer Security Incident Handling Guide (NIST SP 800-61, Cichonski et al. (2012)) describes the security incident management process, which consists of four phases “preparation,” “detection and analysis,” “containment, eradication, and recovery,” and “post-incident activity,” as depicted in Fig. 1. The purpose of the preparation phase is to ensure the SOC has the necessary visibility and potential to detect and respond to security incidents. The preparation stage also includes activities aiming to prevent security incidents altogether, such as malware prevention or user awareness training. The detection and analysis phase consists of responding to security incidents by analyzing, documenting, and prioritizing them and finally notifying the necessary people. The third phase is about containing the security incident by limiting the potential damage the incident may cause, followed by evidence gathering and identifying the attackers’ host. Once the impact of the incident has been limited and the source of the attack has been identified, the threat can be eradicated, and recovery actions can be started (Cichonski et al., 2012). The final stage, post-incident activity, does not contain activities for which the SOC would typically be responsible. Other processes in the SOC can, for example, include data collection and creation, validation, and tuning monitoring rules (Knerler et al., 2022). A SOC can be further split into multiple collections of tightly interlinked functional areas, which according to Knerler et al. (2022), are the following: 1. Incident triage, analysis, and response 2. Cyber threat intelligence, hunting, and analytics 3. Expanded SOC operations 4. Vulnerability management 1https://jyu .finna .fi/. 2https://ieeexplore .ieee .org/. 3https://scholar .google .com/. Computers & Security 135 (2023) 103529 3 J. Forsberg and T. Frantti Preparation activities Detection and analysis Containment, eradication and recovery Post-incident activity Fig. 1. Incident response life-cycle as depicted by Cichonski et al. (2012, p. 21). Metric A Reaction SLA violations Measurement A Reaction time for detections Source 1 SIEM Source 2 EDR Source 3 IDS Measurement B Reaction time for tickets Source 4 Internal Source 5 External Fig. 2. Structure of a metric. 5. SOC tools, architecture, and engineering 6. Situational awareness, communications, and training 7. Leadership and management Larger SOCs may have elements from all of the functional areas above. However, a single operations center does not necessarily need to cover all of the areas to be a functional part of the overall cyber defense capabilities (Knerler et al., 2022). 4. Metrics and measurements The primary purpose of a metric is to measure how well a business process, product, or resource performs so that a business decision can be made Savola (2007). Typically, a metric consists of one or more discrete point-in-time measurements, from which metrics are then derived. Savola (2013)continues that to produce quality metrics for security, the metrics must conform to four fundamental quality criteria: correctness, measurability, meaningfulness, and usability. 4.1. Constructing metrics Metrics can be structured in multiple ways. A metric can simply be a direct relation to a measurement, such as the number of security incidents, or it can be a composite metric constructed by multiple measurements collected from multiple sources. An example of a composite metric is depicted in Fig. 2, in which the metric is constructed from multiple measurements. Measurement A collects the detections generated by security tooling, such as Security Information Event Management (SIEM), Endpoint Detection and Response (EDR), or Intrusion Detection System (IDS). The detections generated by the tools are analyzed by security analysts. Measurement B consists of reports from end-users and other third parties about possible security incidents, such as suspected phishing or security policy violations. Similar to measurement A, the tickets in measurement B are investigated by the analysts. The metric configuration selects the measurements, for which the reaction time by the analysts exceeds the pre-determined value. The reaction time could for example be defined in a Service Level Agreement (SLA) between a SOC and other stakeholders. There are several ways to construct or select metrics that measure the outcome of an activity. A study by Doran (1981) describes a method that utilizes the S.M.A.R.T. method, which is an acronym for specific, measurable, assignable, realistic, and time-related, to select a suitable metric. Brotby and Hinson (2013) introduces the PRAGMATIC method that consists of nine meta metrics (predictive, relevant, actionable, genuine, meaningful, accurate, timely, independent, and cheap), which are essentially scoring criteria for the metric itself. The S.M.A.R.T. and PRAGMATIC methods are more about selecting metrics and which principles should be embraced when a metric is constructed. There are also other methods to construct metrics. For example, the Annex A of the ISO/IEC 27004:2016 (2016) standard summarizes a measurement information model contained in the ISO/IEC 15939 standard and describes how specific attributes related to an entity can be converted into an information product that can be used for conducting business decisions. 4.2. Problems and pitfalls of metrics Metrics can provide valuable insights into the performance of an organization. However, when incorrectly constructed, they can become counterproductive. The metric can be incorrect due to a problem with the raw measurements, a programming error in the algorithm used to derive the metric, or someone is either maliciously or non-maliciously misinterpreting the metric (Brotby and Hinson, 2013). Hauser and Katz (1998)argues that it might also be possible that certain metrics, which are hard to influence by the activities of team members, might lead to a situation where short-term decisions are favored over long-term decisions. A metric may also be precisely wrong, meaning that something is measured with high accuracy, but the metric does not improve the business process it is supposed to improve (Hauser and Katz, 1998). We can also approach the problems with metrics from the quality perspective. An attribute closely related to correctness is unbiasedness and objectivity, meaning the interpretation of the metric should not be influenced by personal beliefs or biases (Savola, 2007). Similarly, reproducibility is closely related to measurability. If a result from a metric cannot be reliably reproduced, the metric may produce an incorrect value. There may also be a problem with the quantity of the metrics. If the metrics are incomplete, it is impossible to understand the overall situation. However, having too many metrics can also be a source of incorrect decisions (Brotby and Hinson, 2013). 4.3. Security operations center metrics Vielberth et al. (2020)concluded that the general level of governance and compliance-related aspects of SOC-related research are Computers & Security 135 (2023) 103529 4 J. Forsberg and T. Frantti immature. There is significant research on security metrics, but the metrics are mostly operational and non-technical. However, they identified some technical metrics, such as false-positive rate, mean time to detect, threat actor attribution, and defensive efficiency. Nonetheless, they failed to show meaningful metrics in several areas, such as automation & orchestration, threat hunting, and detection engineering & validation. Nathans (2014) discussed SOC-related qualitative and quantitative metrics utilized in managing a SOC. He also discussed the importance of vulnerability-related information in detecting potential security incidents by SOCs. The metrics related to the vulnerabilities are categorized as management-related metrics. He presented qualitative metrics, such as the top 10 vulnerable endpoints, and quantitative metrics, including the number of vulnerable endpoints, the number of vulnerabilities per severity, the number of unknown assets, and the time it took to apply a patch that fixed the vulnerability. However, the proposed metrics were not sufficient to be used to demonstrate the technical capabilities of the SOC. Agyepong et al. (2020) presented a framework in which a SOC was split into multiple functions. Each function was measured separately to determine the actual performance of it. These functions included monitoring and detection, analysis, response and reporting, intelligence, baseline and vulnerability, and policies and signature management. The framework proposed that each function should monitor its performance in quantitative and qualitative metrics. The framework did not provide concrete measurement mechanisms upon which organizations can implement the metrics defined in the framework. Keltanen (2019)used results from a customer survey to measure the performance of an outsourced SOC. He ranked the metrics based on the PRAGMATIC method. The resulting score will help evaluate different metrics between one another, making it possible to determine which metric is considered to be the most important (Keltanen, 2019). The study did not present concrete metrics that SOCs could use to measure their performance, instead focused on how the metrics could be constructed. Kokulu et al. (2019) presented a qualitative study on the issues observed by SOC practitioners. The study was based on interviewing eighteen persons working in a SOC. The interviewed analysts mentioned that the metrics selected for measuring the performance were used to measure completely irrelevant things and to demonstrate a false improvement to the upper management rather than the actual performance of the SOC. Onwubiko (2015) presented a framework consisting of metrics used to evaluate SOCs’ performance and determine the return on investment. The framework does not contain a concrete set of metrics, but instead, it provides the top five examples that should be considered, which are the number of incidents, the performance of the cyber operations, the top ten cyber attacks, a summary of policy violations and a summary of privileged user misuse detections. Alahmadi et al. (2022) present a comprehensive qualitative study of SOC analysts’ perspectives on security alarms. The authors state that security alarm validation is a tedious task that can cause alarm burnout and eventually desensitization. They continue that vendors and researchers must be able to make distinctions between types of false-positive alarms. They also argue that researchers have used falsepositive as a metric for evaluating system performance when proposing security tools, seeking few false-positives for optimal performance. In addition to scientific literature, there is a plethora of commercial material that discusses the metrics to measure SOCs. Zimmerman and Crowley (2019) presented seven high-level groups for the metrics. The groups are the health of the data feed, coverage of the monitoring, vulnerability-related metrics, monitoring rules, analysts’ performance, incident handling, risk priorities, and general hygiene. Gartner published a guide (Ahlm, 2021) discussing the industry’s best practices for building and operating a modern SOC, among which a set of fundamental metrics were defined. The metrics mainly focused on quantitative metrics and were grouped into four categories: incident volume, incident detection, incident response, and incident impact. The metrics in these categories, for most parts, are not relevant to the technical performance of the SOC. SANS Institute (Crowley and Pescatore, 2019) surveyed 355 organizations about common and best practices for SOCs. There were questions about which metrics are used in the SOC or measure its performance. The results of the survey conclude that quantitative metrics such as the number of incidents handled and time from detection to containment to eradication are the most common. The three least commonly deployed metrics were “Losses accrued vs. losses prevented,” “Monetary cost per incident,” and “Avoidability of the incident,” all of which are relatively difficult to implement. Out of the metrics mentioned, there were a few metrics that are relevant for measuring the technical performance of the SOC, which are “Threat actor attribution,” “Thoroughness of eradication,” and “Thoroughness and accuracy of enterprise sweeping.” Logsign (2020)groups recommended metrics into two categories, metrics for security operations and metrics related to business requirements. The number of security incidents was raised as the most important metric for security operations. It also depicted metrics, such as the number of alerts per analyst, the number of alerts closed by automation, the number of false-positive alerts, and the average time to detect a security incident. A blog post by Simos and Dellinger (2019) presented some of the key metrics used at the Microsoft SOC. These include time to acknowledge, time to remediate, escalation between tiers, and the number of incidents remediated grouped per response type (manual or automated). The metrics are mostly about measuring the response capabilities of the SOC and are highly relevant to the technical performance of the SOC. In addition to SOC-specific frameworks and methodologies, several publications describe ways of measuring information security on a general level. The Performance Measurement Guide for Information Security (NIST SP 800-55) describes how the organization can create, select and implement metrics for monitoring the state of the security program on an overall level (Chew et al., 2008). The ISO/IEC 27004:2016 standard describes guidelines that can be utilized for measuring the effectiveness of the information security management system (ISO/IEC 27004:2016, 2016). Salmi (2018)surveyed information security metrics implemented in large Finnish corporations. He identified 28 security metrics categorized into either management, operational, or technical metrics. The study contained no metrics that directly relate to the technical performance of a SOC. However, several metrics are closely related to the typical activities of the SOC, such as the business impact of security incidents, characteristics of security incidents, and system vulnerabilities. As a summary of SOC metrics, Table 1describes the top thirty most commonly mentioned metrics in the literature, i.e., the number asterisks depict how many articles the metric was mentioned in. A full list of the metrics observed has been published in Mendeley Data (Forsberg, 2023). In Table 1, one of the metrics was combined into a generalized term instead of having two separate rows for “time to resolution” and “time to incident closure.” Another general level observation was that the terminology does not appear to be consistent in the literature. For example, the metric “mean time to detect” is used to describe at least two different behaviors, the time it takes to react and perform the analysis of the alert (Ahlm, 2021; Agyepong et al., 2020; Crowley and Pescatore, 2019) and time it takes for a SOC to become aware of the incident (Logsign, 2020; Vielberth et al., 2020). Some of the publications did not provide enough information to clearly distinguish between the two (Kokulu et al., 2019; Zimmerman and Crowley, 2019). However, Vielberth et al. (2020)provided a separate metric for the average analysis time and detection time. Thus a conclusion could be reached that the correct definition of time to detect metric would be the time between the initial activity of the adversary and the first detection caused by the activities. This viewpoint is also supported by the Computer Security Incident Handling Guide (NIST SP 800-61, Cichonski et al. (2012)). Computers & Security 135 (2023) 103529 5 J. Forsberg and T. Frantti Table 1 Top 30 metrics in the literature. Chew et al. (2008) ISO/IEC 27004:2016 (2016) Salmi (2018) Vielberth et al. (2020) Nathans (2014) Keltanen (2019) Agyepong et al. (2020) Kokulu et al. (2019) Onwubiko (2015) Alahmadi et al. (2022) Crowley and Pescatore (2019) Ahlm (2021) Logsign (2020) Simos and Dellinger (2019) Zimmerman and Crowley (2019) Numberofsecurityincidents ***** *** *** Mean time to reaction * * * * * * * Number of vulnerabilities * * * * * * * False-positiverate *** ** ** Mean time to detect * * * * * * Mean time to resolution * * * * * * Cost of security incidents * * * * * Detections per category * * * * Mean time to vulnerability remedy * * * * Number of vulnerable devices * * * * Percentage of employees trained * * * Percentage of standard systems * * * Analyst productivity * ** Coverage of vulnerability scanning * * * Downtime due to security incidents * ** Incident avoidability * * * Incidents with business-impact * ** Mean time to containment ** * Mean time to triage * * * Number of incidents per shift * * * Number of monitored assets * * * Number of patched vulnerabilities * * * Number of risk per severity * * * Resolution SLA breaches * * * Severity of security incidents * * * Threat actor attribution * * * Number of automated incidents ** Mean time to escalation ** Quality of eradication ** Reaction SLA breaches * * The most common metric was the total number of security incidents, which was mentioned in eleven of the fifteen publications included in Table 1. This is not a surprise, given the number of incidents is mentioned as a specific metric in both NIST SP 800-53 and ISO/IEC 27004. It is a metric that can easily be collected and used outside of the scope of SOC. Vulnerability-related metrics are also relatively common, with the count of vulnerabilities being mentioned in seven publications. The mean time to vulnerability remediation and the count of vulnerable devices were mentioned in four publications. Operative metrics, such as mean time to reaction, detection, resolution, containment, triage, and escalation, were also commonly mentioned. The publications also contained a few technical performance metrics. They were false-positive rate, threat actor attribution, the number of security incidents closed with automation, and the quality of eradication. The literature review emphasizes the need for a common framework that could be used for the performance evaluation of SOCs globally. The metrics presented in Table 1are scattered broadly, and there appears to be no common group of key metrics used to measure the technical performance of a SOC. The published literature mostly focuses on operational SOC or general security metrics. Although commercial whitepapers provide slightly better technical performance metrics, they fall short in several ways. For example, the lack of proper justification is seen throughout them. Despite the limited scientific research on the subject, several studies have reached a similar conclusion (Agyepong et al., 2020; Keltanen, 2019; Kokulu et al., 2019; Vielberth et al., 2020). The lack of standard technical performance metrics could be attributed to the lack of a sufficiently mature governance model for SOCs, as pointed out by Vielberth et al. (2020). 4.4. Other work on measuring SOCs In addition to the literature mentioned earlier, there are several publications on the ways to measure SOCs. For example, Jacobs et al. (2013), Van Os (2016)and Schlette et al. (2021)are proposing to measure SOCs based on a capability maturity model, Schinagl et al. (2015) approaches the assessment through a set of questions to understand the perceived level of effectiveness and Rosso et al. (2022) introduces a way to measure SOCs performance by injecting simulated attacks into an operational SOC and utilize pre-existing metrics to determine the level of effectiveness. All of these methods suffer from the same fundamental problem: they do not provide concrete metrics a SOC can be measured with. The method proposed by Rosso et al. (2022) succeeds in measuring the technical performance of SOCs, but relies on existing metrics to do so. Some studies discuss methodologies and provide concrete metrics to measure the SOC performance, such as the papers by Shah et al. (2018)and Agyepong et al. (2023), but as they focus on the operational aspects of the SOC, they do not provide ways to measure the technical performance of SOCs. 5. Solution objectives The objective of the solution is the documentation of an approach for creating SOC-related metrics. Additionally, the solution should be used to create three to seven metrics that can either augment existing technical performance metrics or introduce completely new metrics and provide capabilities to measure the technical performance in an Computers & Security 135 (2023) 103529 6 J. Forsberg and T. Frantti Stage 1 Solution objectives Stage 2 Creation framework Stage 2b Metric characteristics Stage 2a Metric requirements Fig. 3. The metric construction stages. Metric creation framework Metric characteristics Target audience and intended use Required measurements Measures to reduce bias and subjectivity Can the characteristics be defined? Valid metric Invalid metric Format for the presentation of the metric Required additional contextual information Interpretation instructions Metric requirements The metric has a clear and well-defined goal The metric can be justified The owner of the metric is well-defined Are the requirements met? Valid metric Invalid metric The metric is tied to a success factor The results are not dependent on third parties The metric is aligned with the quality criteria Yes No Yes No Fig. 4. The metric creation framework. area where previous metrics are incomplete or missing. Based on the literature review, the requirements for the solution are the following: 1. A selection criteria for the creation of metrics should be welldefined. 2. A separate quality criteria for the metrics should be defined and the metrics created with the framework should conform with it. 3. The metrics should be directly associated with a specific SOC function. 4. The metrics should be universal and not tied to a specific technology or an organizational structure. 5. The metrics should be justified either by scientific research, industry standards, or through other means that considerably decrease the subjectivity of the metrics. The justification for the objective 1 is that there does not appear to be an industry-standard framework for creating metrics for SOCs. There are selection criteria such as S.M.A.R.T. (Doran, 1981) and PRAGMATIC (Brotby and Hinson, 2013), but the criteria remain subjective at best. Creating a comprehensive framework for measuring the performance of a SOC is out of the scope of this research, and the focus is placed on enabling the creation of selection criteria that can be used as a part of a comprehensive framework. The justification for the objective 2 is that low quality metrics can produce unwanted results, as discussed in section 4.2. Furthermore, based on the authors’ first-hand experience, many of the metrics used for measuring SOC are of low quality, particularly regarding bias and objectivity. This viewpoint is partially supported by Kokulu et al. (2019). However, the quality aspect of the metrics was not discussed in detail in the SOC-related literature to form a valid conclusion about the insufficient quality. To ensure the resulting metrics are of high quality, quality criteria for SOC metrics should be defined. The justification for the objectives 3 and 4 is the same, which is the universal applicability of the resulting metrics. In practice, this means the metrics should be usable by most SOCs out there, provided that they include the function to which the metric is tied. Finally, the justification for the objective 5 is that the metrics in some situations do not appear to be backed up by scientific research, particularly regarding commercial sources, which means the metrics are purely based on the authority of the author. 6. The metric creation framework This section presents the creation of the design science artifact, i.e., a metric creation framework. It can be used to construct metrics used to measure the performance of a SOC. The artifact is evaluated by utilizing it to create metrics to measure the technical performance of the SOC. The metric construction model is based on requirements defined in two separate stages, as depicted in Fig. 3. The first stage consists of the solution objectives, and the second stage is the metric creation framework that consists of the requirements the metric should conform to and the characteristics that must be describable in the metric documentation, see Fig. 4. A valid metric must fulfill both the metric requirements and the characteristics. The metric requirements are: 1. The metric has a clear and well-defined goal. 2. The owner of the metric is clear. 3. The results are not dependent on third parties. 4. The metric can be justified. 5. The metric is tied to a success factor. 6. The metric is aligned with the quality criteria. A well-defined goal means the metric must be meaningful as per the quality criteria by Savola (2013)and the PRAGMATIC methodology by Brotby and Hinson (2013). The quality criteria within the context of a SOC are shown in Table 2. The metric should also be assignable directly to a function within a SOC. If a third party influences the metric, the metric does not measure only the performance of the SOC but rather the entire chain related to threat detection and incident response. The metrics should also be justifiable by scientific research, industry standards, or through other means documented in the metric descrip- Computers & Security 135 (2023) 103529 7 J. Forsberg and T. Frantti Table 2 The quality criteria for the SOC metrics. Criteria Characteristic Description Correctness Granularity The metric should provide the necessary granularity to tie the metric to a specific function or team within a SOC. Completeness The metric should completely fulfill the goal defined in the metric documentation. If the metric cannot alone fill the goal, the metric should be coupled together with another metric. Objective and unbiased The results should not be influenced by the activities performed by the person setting up the metric and the bias should be minimized to acceptable levels defined in the metric description. Measurability Availability Measurements used to construct the metric must be automatically available in a reliable and consistent format. Reproducibility The metric must be reproducible by different persons across multiple organizations given access to the same measurements. Meaningfulness Impactful The metric must have an impact on the daily activities and it must be capable of showing the progression of development efforts. Clarity The interpretation of the metric must be unambiguous and consistent across the entire lifecycle of the metric. Comparability The result of the metric must aim to be comparable between multiple SOCs even between organizations. Usability Portability The metric must be usable by multiple different SOCs and not be dependent on their size, structure, service model, or parent organization. Controllability The team the metric is used to measure must be capable of keeping the metric value between the expected values. Scalability The metric must be able to behave consistently with low and high volumes of measurements. Presentable It must be possible to present the information the metric is expected to provide visually. tion. If the metrics are justifiable, there is no need to rely on the authority of the source. Establishing a connection between the indicator and the critical success factors (CSFs) is a fundamental requirement for key performance indicators (KPIs) as described in the book by Parmenter (2019). On a practical level, establishing a connection between the CSFs and the KPIs forces organizations to identify critical contributors to organizational performance and thus qualify to be measured with the KPIs. Finally, the metric should conform to the quality criteria, Table 2. If the metric has passed the requirements, it can be constructed. The metric can have different characteristics depending on variables, such as which SOC function it relates to or the stakeholders the metric targets. The fundamental characteristics that should always be defined are the following: 1. Target audience and intended use 2. Measures to reduce bias and subjectivity 3. Required additional contextual information 4. Required measurements 5. Format for the presentation of the metric 6. Interpretation instructions The target audience of the metrics and the intended use must be well defined because there is a mismatch between the evaluation metrics when it comes to SOC managers, SOC analysts, and other technical personnel (Kokulu et al., 2019). If we can reduce bias to a minimum and ensure the metric is objective, we can fulfill the most important quality criterion, correctness. There can also be situations where a counter-metric or other additional contextual information is needed. For example, a SOC may have a metric to measure the number of distinct monitoring rules, which could be used to measure the detection potential of the SOC. However, the high number of monitoring rules does not directly correlate with the performance of the SOC, since if a majority of the security incidents resulting from the monitoring rules are false-positive, the SOC is unlikely to be able to handle them effectively (Alahmadi et al., 2022). Therefore, the metric for the number of monitoring rules should be coupled with the false-positive rate to measure the effective detection potential. As the metrics consist of measurements, the source for the measurement data must be defined along with the format of measurement, the measurement interval, and any other information that affects the measurements or metrics in any way. For example, a metric measuring the mean time to resolution would require each security incident to have two measurements, one to measure the time when the incident was opened and another one to measure the time when the incident has been resolved. Metrics must also be presented in such a way that they clearly and consistently depict the information the metric is supposed to deliver. For example, the metric can be presented numerically, by various charts or time series graphs, or in a text format within a table. The way the metric is presented should provide the person interpreting the metric with the necessary information to make decisions based on the data seen. Interpreting the metrics is important in ensuring the metrics provide valuable insights for the stakeholders. Although the fundamental idea is that the metrics themselves should be self-explanatory, in the practical sense, some metrics can be hard to interpret. The metric documentation should include the expected way to interpret the results. 7. Metrics for security operations center This section describes how the metrics meet the objectives defined in the metric creation framework. Measurements used to construct the metrics have been programmatically generated and presented using open-source tools and Python modules, most prominently Jupyter Notebooks4and a Python graphing library Plotly.5Additionally, scikit-learn, which is a Python module that can be used for data analysis and machine learning algorithm development (Pedregosa et al., 2011), was used to calculate the statistical fitting of a trend line in the form of linear regression. The parameters for creating the data points are adjusted every 100 steps to create variation in the results over time. The source code for the metrics is published in Mendeley Data (Forsberg, 2023). 7.1. Distribution of detections among the UKC Distribution of detections among the Unified Kill Chain (UKC) measures how effective the SOC is in detecting threats in the early stages of the UKC and thus decreases the impact of a security incident. The metric is tied to a function responsible for custom analytics and detection creation, as per the definition by Knerler et al. (2022). If the SOC does not have a team responsible for the function, the metric is not measuring the performance of a SOC but rather the tools the SOC is using. In that case, the metric depends on third parties and thus is not a valid metric for a SOC. Fig. 5depicts a visual representation of the metric with a detection strategy focused on the initial foothold stage. This means that a larger 4https://jupyter .org /about. 5https://github .com /plotly /plotly .py. Computers & Security 135 (2023) 103529 8 J. Forsberg and T. Frantti Fig. 5. Initial Foothold focused detection strategy. (For interpretation of the colors in the figure(s), the reader is referred to the web version of this article.) Fig. 6. Network Propagation focused detection strategy. portion of the detections is within the initial foothold stage, compared to network propagation or the action on objectives stages. Fig. 6depicts the metric with a detection distribution focused on network propagation. Both strategies statistically have a similar amount of detections in the final phase of the kill chain. Thus, both are equally effective in preventing adversaries from achieving their objectives. However, the strategy focusing on the initial foothold stage is better in terms of the metric, as it is more effective at preventing the incidents from traversing forward within the UKC. Fig. 5also shows that the value of the linear regression coefficient decreases over time, meaning the detection strategy shows signs of improvement. By contrast, the trend is increasing in Fig. 6, showing signs of deterioration. Both figures display a large variation at the beginning of the metric, which means the metric is not accurate until there is a sufficient quantity of true-positive detections, but otherwise, the metric conforms with the requirements defined by the framework. 7.2. Number of verifiable monitoring rules The metric Number of verifiable monitoring rules measures the share of monitoring rules that can be automatically or manually verified by executing actions that trigger the rules. The goal of the metric is to improve the verification rate of the monitoring rules, encourage detection engineers, who are responsible for the creation of the detection capabilities, to better utilize threat intelligence as a part of their daily routines, and provide a means for a SOC to demonstrate what attacks they can detect. The metric is tied to a function responsible for custom analytics and detection creation, and it is valid only if the SOC has a team responsible for it. The metric output is not affected by external conditions and meets the clarity and scalability characteristics of the quality criteria. The metric is usable by multiple SOCs, although the interpretation can vary slightly between SOCs. The team responsible for threat detection can