LocVul: Line-Level Vulnerability Localization based on a Sequence-to-Sequence Approach

Highlights LocVul: Line-Level Vulnerability Localization based on a Sequence-to-Sequence Approach Ilias Kalouptsoglou, Miltiadis Siavvas, Apostolos Ampatzoglou, Dionysios Kehagias, Alexander Chatzigeorgiou •A Sequence-to-Sequence approach is proposed for line-level vulnerability localization. •The Sequence-to-Sequence approach outperforms XAI-based methods in vulnerability detection. •LLM hallucinations are mitigated by matching generated lines to real ones via cosine similarity. LocVul: Line-Level Vulnerability Localization based on a Sequence-to-Sequence Approach Ilias Kalouptsogloua,b,∗, Miltiadis Siavvasa, Apostolos Ampatzogloub, Dionysios Kehagiasa, Alexander Chatzigeorgioub aCentre for Research and Technology Hellas/Information Technologies Institute, 6th km Charilaou-Thermi Rd, Thermi, 57001, Thessaloniki, Greece bUniversity of Macedonia/Department of Applied Informatics, Egnatia 156, Thessaloniki, 54636, Thessaloniki, Greece Abstract Context: The development of secure software systems depends on early and accurate vulnerability identification. Manual inspection is a time-consuming process that requires specialized knowledge. Therefore, as software complexity grows, automated solutions become essential. Vulnerability Prediction (VP) is an emerging mechanism that identifies whether software components contain vulnerabilities, commonly using Machine Learning models trained on classifying components as vulnerable or clean. Recent explainabilitybased approaches attempt to rank the lines based on their influence on the output of the VP Models (VPMs). However, challenges remain in accurately localizing the vulnerable lines. Objective: This study aims to examine an alternative to explainability-based approaches to overcome their shortcomings. Specifically, explainability-based methods depend on the type and accuracy of the file or function-level VPMs, inherit possible misleading patterns, and cannot indicate the exact code snippet that is vulnerable nor the number of vulnerable lines. Method: To address these limitations, this study introduces an innovative approach based on fine-tuning Large Language Models on a Sequence-to-Sequence objective to directly return the vulnerable lines of a given function. The method is evaluated on the Big-Vul dataset to assess its capacity for fine-grained vulnerability detection. Results: The results demonstrate that the proposed method significantly outperforms the explainability-based baseline both in terms of accuracy and cost-effectiveness. Conclusions: The proposed approach marks a significant advancement in automated vulnerability detection by enabling accurate line-level localization of vulnerabilities. Keywords: Software Security, Vulnerability Detection, Large Language Models, Sequence-to-Sequence Models, Self-Attention ∗Corresponding Author. Email addresses: [email protected], [email protected] (Ilias Kalouptsoglou), [email protected] (Miltiadis Siavvas), [email protected] (Apostolos Ampatzoglou), [email protected] (Dionysios Kehagias), [email protected] (Alexander Chatzigeorgiou) 1. Introduction1 Security is an important characteristic of the Software Development Life-Cycle (SDLC)2 according to the International Standard on Software Quality ISO/IEC 25010 [1]. How-3 ever, in practice it is usually considered an afterthought and is not addressed until the4 final stages of the overall SDLC. Software enterprises often neglect enhancing the secu-5 rity and the resilience of their products on the altar of fast delivery [2] or due to lack6 of security expertise [3]. As a result, the number of reported vulnerabilities continues to7 rise each year, according to the published Common Vulnerabilities and Exposures (CVE)8 records [4].9 Software vulnerabilities are weaknesses within the source code that can be exploited10 by one or more external threats [5]. They are often caused by a small number of common11 programming errors made by developers during the coding phase, but their exploitation12 can lead to devastating consequences in terms of data privacy and system confidentiality,13 integrity, and availability as well as serious financial damage. For instance, one of the14 most notable critical vulnerabilities, the Log4Shell vulnerability was found in a widely-15 used open-source logging library called Log4j. It can allow an attacker to control the16 remote systems, execute malware and leak sensitive information (e.g., passwords), among17 others [6].18 To avoid similar circumstances, it is important for software houses to adopt proper19 security assessment and vulnerability detection solutions from the early phases of the20 SDLC [7]. However, the continuous increase in the number of security breaches and21 vulnerabilities exploited, especially considering the fact that many vulnerabilities remain22 undetected for years (e.g., Heartbleed remained unnoticed for two years and Log4Shell23 for eight years) [8], demonstrates the insufficiency of the tools already in use. Hence,24 there is a strong need for innovative and accurate vulnerability detection techniques.25 1.1. Background26 Existing vulnerability detection techniques are divided in traditional and data-driven27 categories. The former include static and dynamic analysis whereas the latter leverage28 Machine Learning (ML) to perform vulnerability prediction (VP). Static analysis tools29 (e.g., SonarQube, Coverity, Checkmarx, etc.) [9] identify buggy lines of code based on30 rules and code patterns defined by experts. However, static code analyzers often miss31 certain types of vulnerabilities since not all vulnerabilities can be modeled by rules, and32 every analyzer has its own detection rules and supports different vulnerability categories33 [10]. Moreover, static analysis is too broad producing many false positives, overwhelming34 security teams with a large amount of false alarms [11],[12]. On the other hand, dynamic35 testing techniques, such as fuzzing and penetration testing, analyze running software36 and simulate attacks to identify weaknesses. However, dynamic testing still produces37 both false positives and false negatives, cannot be applied to every component, and has38 difficulty on tracing the vulnerability in the code [13].39 As regards the data-driven approach, VP involves constructing vulnerability predic-40 tion models (VPMs), which are ML models trained on software attributes extracted41 from the source code of files or functions/methods in order to predict the existence of42 vulnerabilities in them. The VP-related research evolved from the construction of ML43 models based on software metrics (e.g., complexity, cohesion, etc.) [14],[15],[16],[17] to44 building Deep Learning (DL) models capable of learning patterns in the text of the45 2 source code, similarly to Natural Language Processing (NLP) tasks [18],[19],[20]. Typi-46 cally text mining attempts represent the source code as sequences of tokens, where each47 token is encoded with an embedding vector [21],[22], or as text-rich graphs, such as the48 Code Property Graphs (CPGs) [23], which are fed to Graph Neural Networks (GNNs)49 to extract a graph embedding of the source code [24],[25],[26]. Recently, with the break-50 through of the Transformer architecture [27], the focus of the VP-related literature has51 shifted to transfer learning approaches. Several studies employ pre-trained models to52 build Large Language Models (LLMs)-based VPMs [28],[29],[30],[31]. It can be argued53 that LLMs have dominated the VP field, but, despite being considered as a promising54 solution for addressing open issues in the literature [32], the LLM-based VP solutions55 still face important challenges [33].56 1.2. Motivation for line-level vulnerability detection57 A major challenge in the domain that hinders the adoption of ML-based VP solutions58 in practice is the level of granularity at which the predictions are made. Most studies59 in the literature are coarse-grained as they predict whether there are vulnerabilities at60 file or function level of granularity [32]. Such predictions could be useful for the devel-61 opment teams to prioritize their inspection and fortification efforts to specific software62 components, but are still far from achieving complete vulnerability detection, since they63 lack the ability to present the specific lines of the components containing the vulnera-64 bilities, let alone the vulnerability categories [34]. Therefore, coarse-grained predictions65 discourage software engineers from using such tools. It can be argued that the lower the66 granularity, the more actionable the results and the easiest manual inspection, but it is67 more difficult to build accurate mechanisms.68 To clearly demonstrate the benefit from fine-grained VPMs, Figure 1 illustrates the69 difference between the outputs of function-level and line-level approaches using a buffer70 overflow example. In this example, processInput function receives user input and copies71 it into a fixed-size buffer without verifying the input’s length. This operation introduces72 a buffer overflow vulnerability, which can result in overwriting adjacent memory locations73 and potentially lead to arbitrary code execution. In Figure 1, the entire processInput74 function is highlighted with an outer, thicker red rectangle, representing the output of a75 typical function-level VPM, which flags a whole function as potentially vulnerable. This76 narrows the inspection focus (e.g., by a human reviewer) from the entire class to this77 specific function. In contrast, the inner, thinner red rectangle highlights only the lines78 directly responsible for the vulnerability, reflecting the output of a line-level prediction79 approach. This finer-grained localization provides developers with much more actionable80 guidance, allowing them to directly identify and remediate the exact code statements81 that are vulnerable, rather than needing to manually inspect every line in the function.82 1.3. Motivation for the proposed LocVul approach to Localize Vulnerabilities83 The vast majority of fine-grained solutions in the VP-related literature employ ex-84 plainable artificial intelligence (XAI) to interpret function or file-level predictions and,85 thereby, localize vulnerabilities [33]. More specifically, current mechanisms that per-86 form VP at a low granularity level focus on the adoption of XAI techniques to extract87 the features’ importance in the predictions made by VPMs [35]. A prime example of88 such solutions is the LineVul [36] study, in which, Fu et al. [36] proposed fine-tuning a89 3 Figure 1: A motivating example of line-level vulnerability detection compared to coarser-grained predictions. pre-trained on code LLM (i.e., CodeBERT[37]) for function-level predictions and then90 extracting its Self-Attention [27] weights to get line-level predictions, achieving quite ac-91 curate VP at line-level of granularity. From this point on, line-level VP will be referred92 to as Vulnerability Detection (VD).93 However, such XAI-based solutions heavily depend on the function-level prediction94 models. Thus, they can only be applied on top of the VPMs, limiting the choices and95 possible improvements of VPMs of different kinds. In addition, VPMs often make a96 correct prediction based on irrelevant or even spurious correlations among tokens in the97 input sequence, which the VPMs erroneously associate with the existence of vulnerabil-98 ities [38],[39]. Therefore, XAI is often prone to fail in correctly identifying vulnerable99 lines. Moreover, such XAI-based solutions assign Attention scores to tokens and, based100 on them, assign importance scores to lines. Thus, although they can rank the lines pri-101 oritizing the reviewer’s (e.g., developer) efforts to search for vulnerabilities starting from102 the higher ranked lines, they are not able to directly showcase which or how many of the103 lines are the vulnerable ones (i.e., there is no indication of when to stop the inspection).104 4 Figure 2 is provided to illustrate the aforementioned limitations of the XAI-based VD as105 opposed to the proposed solution for Localizing Vulnerabilities (LocVul).106 Figure 2: A motivating example of the proposed LocVul method compared to XAI-based methods. Specifically, Figure 2 (A) presents the source code of a given function included in107 the dataset used, which contains a weak cryptography vulnerability. The exact location108 of the vulnerability is highlighted with red lines. As can be seen in Figure 2 (B), the109 explainability-based technique (e.g., Self-Attention mechanism) returns a list of the lines110 of the function sorted from the most likely to be vulnerable to the least likely. In contrast,111 as can be seen in Figure 2 (C), LocVul returns the exact set of lines it considers vulnerable,112 as highlighted in gray. LocVul response can then be filled in by the remaining lines of113 the function. Thus, LocVul’s response facilitates the inspection process, since the code114 reviewer would have an indication of the number and the exact location of the vulnerable115 lines. Moreover, one can see in Figure 2 (B) that there are several lines before the116 vulnerable ones in the list proposed by the explainability-based method, which indicates117 that although the VPM at the function level has correctly identified the vulnerable118 function, it has also emphasized features that are not related to the vulnerability (i.e.,119 spurious data associations).120 1.4. Novelty and contribution121 In this study, we propose an alternative approach named LocVul to perform line-122 level vulnerability prediction (i.e., vulnerability detection) that uses two distinct models123 for function-level and line-level prediction respectively, managing to overcome the afore-124 mentioned limitations. Specifically, in the proposed approach, the line-level model (i) is125 independent from the function-level VPM and, therefore, can be implemented on the top126 of metrics, text, graph-based or ensemble VPMs; (ii) does not propagate the error of the127 spurious patterns learned by the VPMs; and (iii) first returns a string with the specific128 lines it considers vulnerable and then returns a ranked list of lines so that the inspector129 has both an indication of the exact lines identified as vulnerable and, complementarily,130 5 a list of all lines sorted by their likelihood of being part of the vulnerability. Moreover,131 LocVul achieves higher accuracy and better cost-effectiveness than XAI-based methods.132 LocVul is based on a Sequence-to-Sequence (Seq2Seq) approach, where a Text-to-Text133 Transformer-based model [40] is trained on the task of extracting the vulnerable lines from134 functions that are predicted as vulnerable by a text mining-based VPM. More specifically,135 we first fine-tune a pre-trained model (i.e., CodeBERT) on the objective of classifying136 functions as vulnerable or not and, also, we fine-tune a pre-trained Sequence-to-Sequence137 model (i.e., CodeT5[41]) in the vulnerable lines extraction objective. Subsequently, we138 execute the classification model on unseen data to predict vulnerable functions and, then,139 we apply the Sequence-to-Sequence model on the predicted as vulnerable functions to140 get the vulnerable code snippets that are included in them.141 To the best of our knowledge, LocVul is the first Sequence-to-Sequence VD approach,142 which manages not only to overcome the limitations of the XAI-based approaches, but143 also to clearly surpass their detection accuracy. Additionally, this study observes that the144 Sequence-to-Sequence model produces some imprecise outputs, which could be an im-145 portant limitation of this approach. In particular, we observed that the developed model146 often generates similar but not identical vulnerable lines as those present in the analyzed147 function, which is considered an issue related to LLMs hallucinations [42]. Therefore,148 the present study proceeds by proposing also a solution that eliminates this problem149 by replacing the not-exact matches with the most similar lines in the original function150 (see Section 3.3). Overall, the contributions of the current study can be summarized as151 follows:152 •A novel non-XAI-based VD mechanism is proposed based on a Sequence-to-Sequence153 model: We propose a two-stage approach where (i) a CodeBERT-based classifier154 is fine-tuned to identify vulnerable functions, and (ii) instead of explaining the155 function-level predictions, a CodeT5-based Sequence-to-Sequence model is fine-156 tuned to directly extract vulnerable lines from given vulnerable functions. The157 approach is trained and evaluated in the Big-Vul dataset.158 •Superior than state-of-the-art results are achieved in terms of accuracy and cost-159 effectiveness: Our method significantly improves various evaluation metrics over the160 Self-Attention XAI baseline in identifying vulnerable lines in vulnerable functions.161 •A method to mitigate LLM hallucinations in VD is proposed: We propose a line162 replacement post-processing step using cosine similarity between the embedding163 vectors of the tokenized lines to match generated lines to those present in the164 function, mitigating “hallucinated” outputs from CodeT5.165 •A replication package of our LocVul implementation is published publicly: We166 provide our source code, models, and guidelines to reproduce the results [43].167 The rest of the paper includes Section 2, which summarizes the related work in the168 literature and Section 3 that provides a detailed description of the proposed approach.169 Subsequently, Section 4 sets up the experimental design, while the results of our exper-170 imental analysis are presented in Section 5. Section 6 discusses the insights and lessons171 learned from the analysis, and finally, Section 7 provides the conclusions of the study172 and suggestions for future work.173 6 2. Related work174 This section presents previous studies in the field of ML-based vulnerability prediction175 (VP). Initially, we describe notable studies regarding VP in file and function level, and176 subsequently, we discuss more fine-grained techniques.177 2.1. Coarse-grained vulnerability prediction178 Initial attempts at ML-based VP used software metrics as indicators of the exis-179 tence of vulnerabilities. Shin and Williams [14],[44] were among the first to explore the180 potential of complexity metrics to differentiate between vulnerable and non-vulnerable181 functions. Chowdhury and Zulkernine [45] then presented a VP paradigm by using com-182 plexity, cohesion, and coupling metrics and training several ML models. Furthermore,183 Kalouptsoglou et al. [15] utilized a variety of statically extracted software metrics to184 predict vulnerable files using the Multi-Layer Perception (MLP) classifier.185 Moreover, text mining-based studies were presented in the literature, which were ini-186 tially based on the Bag of Words (BoW) text representation format, where each software187 component is a table, its words are the table’s columns, and the frequencies of the words188 are the columns’ values [18],[46],[47]. Comparative studies [8],[48] have shown that text189 mining-based VP is more accurate than software metrics-based VP, managing to capture190 more complex patterns in the source code. Text mining-based approaches evolved from191 the more simplistic BoW to the sequences of tokens representation format, which repre-192 sents the source code of a file or function as a sequence of words (i.e., tokens). The tokens193 sequence is then transformed to a sequence of embedding vectors [22] using algorithms194 such as word2vec [49] and fastText [50]. In particular, Dam et al. [19] employed a Long195 Short-Term Memory (LSTM) model to automatically predict vulnerable files. In addi-196 tion, Li et al. [21] proposed VulDeePecker that utilizes a Bidirectional LSTM to identify197 vulnerable code on tokens encoded with word2vec embeddings.198 Furthermore, Bilgin et al. [24] proceeded with the extraction of the Abstract Syntax199 Tree (AST) of C/C++ methods and the utilization of ML models to classify them to200 vulnerable and non-vulnerable ones. Next, Zhou et al. [25] proposed the Devign model201 based on graphical source code representations (i.e., ASTs, control and data flow graphs)202 and leveraged a GNN to extract the graph embeddings of C/C++ functions, which203 then were classified to vulnerable and non-vulnerable ones. Nonetheless, Chakraborty204 et al. [26] revealed several limitations of the existing VP datasets and approaches and205 proposed a CPG [23] and GNN-based methodology managing to outperform Devign.206 Recently, after the breakthrough of the Transformer architecture [27], several studies207 have employed transfer learning for VP leveraging the vast knowledge and the context-208 aware patterns encapsulated in the pre-trained LLMs [33]. To this end, Bagheri et209 al. [30] compared traditional embedding techniques (i.e., word2vec and fastText) to the210 embeddings extracted from the Bidirectional Encoder Representations from Transformers211 (BERT) [51] in their capacity to classify code snippets in vulnerable and non-vulnerable212 ones. Furthermore, Kalouptsoglou et al. [29] fine-tuned three popular LLMs and their213 pre-trained on code variants observing their superiority compared to traditional text214 mining-based methods and also the superiority of the code variants. They also observed215 an advantage of the encoder-only Transformer-based models in VP.216 Moreover, several empirical studies conducted by researchers compared various pre-217 trained models in the task of VP [52],[53]. In addition, Kim at al. [54] fine-tuned BERT to218 7 construct their VPM called VulDeBERT managing to outperform VulDeePecker, while219 Hanif et al. [55] pre-trained on code a RoBERTa [56] model, added an MLP and a220 Convolutional Neural Network (CNN), and fine-tuned it on vulnerability-related C/C++221 data for VP, outperforming both Recurrent Neural Networks (RNN)-based and GNN-222 based approaches.223 2.2. Fine-grained vulnerability detection224 In an attempt to provide fine-grained vulnerability predictions, Hin et al. [57] em-225 ployed supervised statement-level training. In particular, they presented a GNN-based226 model, namely LineVD, which classifies each line of the predicted as vulnerable func-227 tions into vulnerable or not. Instead of classifying every line, Li et al. [58] proposed228 the IVDetect model following an interpretable (i.e., explainable) ML-based approach.229 Specifically, they represented the source code in Program Dependency Graphs (PDGs)230 [59], performed function-level VP, and then interpreted the model’s decisions using the231 GNNExplainer [60] method that derives interpretation sub-graphs. Those sub-graphs232 correspond to hotspots in the source code that are more relative to the identified vul-233 nerabilities. Similarly, LineFlowDP [61], which employed PDGs to represent source code234 and GNNs to perform file-level prediction of defects (i.e., general bugs or malfunctions),235 leveraged GNNExplainer [60] to explain the predictions and localize defects.236 Nonetheless, Fu et al. [36] emphasized on the limitation of graph-based solutions,237 such as IVDetect [58], in producing truly fine-grained interpretations, since sub-graphs238 contain a lot of lines of code. To this end, they proposed LineVul [36] by leveraging239 the capacity of the pre-trained CodeBERT [37] model. Specifically, they represented240 source code in token sequences, they fine-tuned CodeBERT for the binary classification241 task of VP, and then, they extracted the token scores through XAI (i.e., through Self-242 Attention). By summing the token scores per line, LineVul can rank lines from most243 likely to be vulnerable to least likely. In LineVul study, several XAI techniques were244 examined concluding that the Self-Attention mechanism of the Transformer model is the245 optimal explainability choice. It is also worth noting that all of them clearly outperformed246 the CppCheck [62] static code analyzer. Furthermore, DeepLineDP [63] used Recurrent247 Neural Networks (RNNs) to perform file-level defect prediction on sequences of source248 code tokens, and then, employed an Attention layer [64] to explain the predictions, and249 therefore, extract the tokens that contribute the most to the prediction of the defective250 files.251 However, Cheng et al. [38] observed poor performance of several XAI-based ap-252 proaches in localizing vulnerabilities due to the existence of irrelevant features learned253 by VPMs, the susceptibility of VPMs to perturbations not related to vulnerabilities, and254 the ineffectiveness of the explainers in choosing meaningful characteristics. In addition,255 Sotgiu et al. [39] investigated the capacity of explainability techniques to find the features256 that contribute most to ML model decisions. Their study used an XAI method called257 SHapley Additive exPlanations (SHAP) to highlight important issues in model construc-258 tion that obstruct its ability to identify vulnerabilities. Specifically, they observed that a259 Transformer-based VPM tends to learn spurious correlations among data. For instance,260 the models often emphasize tokens, which should not have a discriminating role in pre-261 diction (e.g., special characters of programming languages). They further observed that262 the top-10 tokens, which led to a negative or positive decision, were the same for both263 classes. Therefore, VD through explainability often leads to incorrect findings.264 8 Top-KAccuracy is defined as x N×100%, where x is the number of vulnerable functions496 whose at least one line is vulnerable in the top-Kof the list, and Nthe total number497 of vulnerable functions. It measures the percentage of functions for which the model498 detects at least one truly vulnerable line in the top-Klines it considers most likely to be499 vulnerable.500 Similarly, we report the average value of the Top-KPrecision (P@K) and Top-501 KRecall (R@K) metrics across all functions. The former stands for the actual vul-502 nerable lines detected in the top-Klines proposed by the model and is computed as503 Number of vulnerable lines in K Total number of lines in K. The latter, which is equal to Number of vulnerable lines in K Total number of vulnerable lines ,504 computes the truly vulnerable lines of the top-Klines proposed by the model among the505 total of vulnerable lines in a function [10].506 Nonetheless, Top-KPrecision and Top-KRecall are indifferent to the order of vul-507 nerable lines in the top-Kranking. They depict the number of vulnerable lines within508 the top-K, regardless of their specific order. Therefore, we utilize Mean Reciprocal Rank509 (MRR) and Mean Average Precision (MAP) at K, which are two rank-aware metrics.510 Specifically, MRR@K is a ranking quality metric, which is equal to the arithmetic511 mean of the Reciprocal Ranks (RR) across all functions, where a RR is the inverse of512 the position of the first vulnerable line. MRR@K is calculated as 1 NPN i=1 1 ranki, where513 Nrefers to the total number of vulnerable functions and rankiis the position of the first514 vulnerable line for function iin the top-Kresults [79]. If no vulnerable lines are found,515 MRR is equal to zero.516 Moreover, MAP@K is the mean of the Average Precision (AP) at Kof all functions517 analyzed. It considers the number of vulnerable lines in the top-Klist and their position518 in the list. First, we compute the AP per function by averaging the precision at each519 position of vulnerable lines in the top-Kranking list. Particularly, MAP@K is calculated520 as 1 NPN n=1 AP@K, where N presents the total number of vulnerable functions, and521 AP@K is calculated as 1 MPK k=1 P@k×rel(k), where Mrefers to the number of vulnerable522 lines in the top-Kresults for a specific function, Krefers to the selected cutoff point,523 P@k is the Top-kPrecision, and rel(k) equals 1 if the line at position kis vulnerable and524 0 if not [79].525 Since such a VD tool would only be useful if it managed to rank the vulnerable lines526 of the analyzed functions at the top of the recommended list of lines, we have to choose a527 relatively small value of K. Therefore, we choose K= 10 to compute the aforementioned528 metrics, following the example of prior studies [36],[58].529 In addition, to measure the cost-effectiveness of the proposed approach, we use metrics530 indicative of the effort required to achieve a sufficient line-level VD performance, First,531 we employ one function-based measurement called Initial False Alarms (IFA) [36],[63],532 which counts how many false alarms (i.e., non-vulnerable lines) occur before the first533 truly vulnerable line in a function. The lower the IFA, the less effort and time is spent534 by the code reviewer in inspecting non-vulnerable lines.535 Furthermore, we also use Effort@K%Recall and Recall@K%LOC to measure cost-536 effectiveness by considering the entire dataset under test as a whole [36],[63]. In partic-537 ular, Effort@K%Recall measures how much effort, expressed in lines of code (LOC), is538 required to identify the K% of the actual vulnerable lines of the entire testing set. The539 lower the value of Effort@K%Recall, the less effort is required for the code reviewer to540 find the K% of vulnerable lines. Moreover, Recall@K%LOC calculates the number of541 15 true vulnerable lines found by making an effort equal to KLOC inspected. The higher542 the Recall@K%LOC the more vulnerable lines are found for a fixed amount of effort.543 Similarly to [36], we use the Effort@20%Recall and the Recall@1%LOC.544 Finally, to ensure that our analysis is not affected by randomness in data shuffling,545 model training, and other computational processes, we repeat all experiments ten times546 using a different random seed each time. We thus collect ten different values for each547 evaluation metric and report their average.548 4.5. Selection of baseline approach549 To demonstrate the efficacy of LocVul in line-level VD, we compare it with the current550 state-of-the-art. Specifically, during the evaluation of our experimental analysis presented551 in Section 5, we conduct a one-to-one comparison with the most notable existing method552 used for VD, as identified through our literature review in Section 2.553 It is well-known in the literature that XAI techniques have been widely adopted to554 address the challenge of VD at line-level of granularity [33],[35],[38]. By inspecting the555 related studies described in Section 2, we observed that graph-based methods, such as556 IVDetect [58], utilize the GNNExplainer [60] to interpret the predictions and, therefore,557 localize the vulnerabilities. However, although they achieve high coarse-grained accu-558 racy, they are disadvantaged in line-level detection compared to text mining-based XAI559 approaches [36],[38].560 Such a text mining-based methodology, which managed to outperform IVdetect lever-561 aging XAI is the one proposed by Fu et al. namely LineVul [36]. They first fine-562 tuned the CodeBERT model in VP, and then used the Self-Attention mechanism of the563 Transformer-based model to explain its predictions. More specifically, LineVul summa-564 rizes the Attention scores of the tokens included in the vulnerable functions to calculate565 line-level scores and, subsequently, ranks the lines of the functions from the most likely566 to be a vulnerable line to the least likely.567 In addition, they compared Self-Attention against various other XAI techniques ap-568 plied on textual code representations. The results presented in LineVul study [36],569 demonstrated the superiority of Self-Attention among XAI-based techniques, since it570 clearly outperformed all of Layer Integrated Gradient (LIG) [80], Saliency [81], DeepLift571 [82],[83], DeepLiftSHAP [84], and GradientSHAP [84] approaches in identifying vulnera-572 ble lines. They also showed a great superiority of Self-Attention compared to a traditional573 (non-XAI based) approach, the static code analysis approach, through the CppCheck [62]574 analyzer. Hence, it can be argued that the Self-Attention mechanism has emerged as the575 cutting-edge approach for the detection of fine-grained vulnerabilities.576 Accordingly, we choose the Self-Attention-based explainability method as our baseline577 for comparison. We do not just report the results presented in the study by Fu et al. [36],578 but we proceed with implementing it from our own (i.e., replicating their methodology)579 to avoid bias in the results due to the potential inconsistency in the implementation580 settings of the compared approaches (i.e., Self-Attention and our LocVul Detector). In581 this way, we can also evaluate Self-Attention using a more broad and representative set582 of evaluation metrics than those presented in LineVul study [36]. Moreover, it is not clear583 in [36] whether the localization results were computed using only the true positives or584 all the predicted as vulnerable samples, with the latter being what we consider to be the585 approach corresponding to a real-world scenario.586 16 In particular, we implement the entire explainability-based methodology that lever-587 ages the Self-Attention mechanism of CodeBERT to localize the vulnerable lines in the588 functions based on the importance of the different tokens of the input in the function-level589 prediction. During the development of this implementation, we consulted the LineVul590 replication package [85]. From now on we will refer to this baseline as the Self-Attention591 approach.592 5. Results593 In this section, the results of our entire analysis are presented. Specifically, we present594 the evaluation results of the proposed Seq2Seq approach (i.e., LocVul) and we compare595 it against the Self-Attention approach that we selected as our baseline (see Section 4.5).596 Regarding the function-level VPM (i.e., LocVul Classifier), which is used as a fisrt step597 for both the Seq2Seq and Self-Attention approaches during our experimental study, the598 CodeBERT-based model achieves a high predictive performance in VP. Specifically, it599 identifies vulnerable functions with 99.04% Accuracy, 95.62% Precision, 86.88% Recall,600 and 91.04% F1-score. Regarding line-level VD, we evaluate both the proposed Seq2Seq601 model (i.e., LocVul Detector) and the state-of-the-art Self-Attention approach in their602 capacity of localizing vulnerabilities. To this end, we provide a detailed analysis with603 respect to the two RQs that we defined in Section 4.1.604 5.1. RQ1Accuracy of LocVul for line-level vulnerability detection605 An early sign that the Seq2Seq-based LocVul Detector has high detection accuracy606 is the fact that, for around 60% of vulnerable functions, the set of lines generated by the607 model exactly matches the ground truth vulnerable lines. Therefore, even if we ignore608 the recommended line ranking lists, LocVul manages to identify the exact vulnerable609 segment in 60% of the vulnerable functions. Nevertheless, to answer RQ1, we evaluate610 the accuracy of LocVul and we compare it against Self-Attention using proper evaluation611 metrics (see Section 4.4). Initially, we calculate the Top-10 Accuracy of the LocVul612 Detector to measure how often it manages to detect one vulnerable line in the 10 first613 lines it suggests for inspection. We then evaluate LocVul in terms of Top-10 Precision614 and Top-10 Recall to also consider the number of lines detected in the 10 first lines.615 Figure 4 presents the values of these metrics for LocVul compared to Self-Attention.616 As can be seen in Figure 4, the LocVul Detector achieves a Top-10 Accuracy equal to617 82.8%, which is 11.4% higher than the 71.4% of Self-Attention, showing that the proposed618 model detects at least one vulnerable line in much more cases than the baseline. In619 addition, the Top-10 Precision of LocVul is 26.9% compared to 19.0% of Self-Attention,620 a result that indicates the superiority of LocVul in identifying multiple truly vulnerable621 lines in the top-10 of the lines it suggests as vulnerable. Specifically, LocVul’s Top-10622 Precision of 26.9% shows that the model identifies on average 2.69 actual vulnerable lines623 in the top-10 recommended lines. Moreover, a Top-10 Recall equal to 79.0%, which is624 much higher than Self-Attention’s 57.7%, highlights the ability of LocVul to detect many625 of the function’s vulnerable lines in the top-10 ranking.626 At this point, we should mention that the results obtained with the developed Self-627 Attention approach for VD are slightly different from those reported in [36]. In particular,628 Top-10 Accuracy is higher by 6.4%. This can be attributed mainly to the fact that, in629 17 (a) Comparison of Top-10 Accuracy (A@10). (↗) Higher A@10 - Better. (b) Comparison of Top-10 Precision (P@10). (↗) Higher P@10 - Better. (c) Comparison of Top-10 Recall (R@10). (↗) Higher R@10 - Better. Figure 4: Top-10 Accuracy, Precision, and Recall of LocVul compared to the baseline Self-Attention method for line-level vulnerability detection. our implementation, we remove the functions that are labeled as vulnerable but have630 incomplete line-level labels (i.e., they have no information on which are the vulnerable631 lines). Those samples act as noise in the line-level dataset and are also a barrier to632 training the Seq2Seq model. Therefore, they have to be removed from both training and633 testing sets.634 Furthermore, to evaluate the accuracy of our approach in a rank-aware manner by635 considering the order of the lines proposed as vulnerable by the model, we compute636 the Mean Reciprocal Rank (MRR) and the Mean Average Precision (MAP) at top-10637 ranking list. Figure 5 shows the results for LocVul and Self-Attention approaches in638 terms of MRR@10 and MAP@10 in the left and right bar charts, respectively. Our639 Seq2Seq-based LocVul approach manages to outperform the XAI-based Self-Attention640 approach by almost double the score in terms of MRR@10 and [email protected] In particular, the LocVul Detector achieves MRR@10 equal to 79.4%, which is 35.8%642 higher than Self-Attention’s 43.6% leading to the conclusion that it is able not only643 to detect a vulnerable line in more functions than XAI techniques, but also to detect644 it earlier in the ranking list of lines. In addition, MAP@10 of the LocVul Detector is645 equal to 79.2% in contrast to Self-Attention, which has MAP@10 equal to 41.1%. This646 observation demonstrates the enhanced capability of LocVul to detect more vulnerable647 lines higher up in the ranking list compared to the Self-Attention mechanism.648 Although a clear difference in the values of the accuracy-related evaluation metrics649 between the two approaches is observed, we apply a statistical test to support further650 our findings. In particular, we conduct the Wilcoxon Signed-Rank Test [86] to check651 whether the paired accuracy scores, obtained from ten different random seed values, dif-652 fer significantly between LocVul and Self-Attention. We repeat this analysis for all of653 the Top-10 Accuracy, Precision, Recall, MRR@10, and MAP@10 metrics. Considering654 as the null hypothesis that LocVul is not more accurate than Self-Attention for each655 metric, the Wilcoxon test returns p-values of 0.00098 across all metrics, which are well656 below the 0.05 significance threshold. Therefore, we reject the null hypothesis and state657 18 (a) Comparison of Top-10 Mean Reciprocal Rank (MRR@10). (↗) Higher MRR@10 - Better. (b) Comparison of Top-10 Mean Average Precision (MAP@10). (↗) Higher MAP@10 - Better. Figure 5: Top-10 Mean Reciprocal Rank and Mean Average Precision of LocVul compared to the baseline Self-Attention method for line-level vulnerability detection. that LocVul is significantly more accurate than Self-Attention. Hence, we can argue that:658 659 The LocVul approach, which extracts vulnerable lines from the source code of vulnerable functions using a code-aware Seq2Seq LLM, specifically CodeT5, achieves high accuracy in vulnerability detection, outperforming the baseline Self-Attention explainability method. 660 5.2. RQ2Cost-effectiveness of LocVul for line-level vulnerability detection661 In RQ2, we investigate the cost-effectiveness of the proposed VD approach. In other662 words, we evaluate the predictive performance of LocVul with respect to the effort re-663 quired to achieve this performance. To this end, we compute the cost-effectiveness evalua-664 tion metrics, which are described in Section 4.4. On the one hand, we evaluate LocVul by665 calculating the IFA metric that evaluates cost-effectiveness by counting the false alarms666 that one has to inspect until finding one truly vulnerable line in a function. On the other667 hand, we evaluate LocVul using the Effort@20%Recall to measure the effort required to668 find the 20% of the vulnerable lines in the entire testing set, while Recall@1%LOC is669 used to find the truly vulnerable lines detected by inspecting a fix amount of LOC (i.e.,670 the 1% of the total LOC) in the testing set. Figure 6 presents the values of these metrics671 for the proposed LocVul approach compared to the baseline Self-Attention approach.672 In particular, Figure 6a presents the IFA values of the compared approaches. Similarly673 to previous studies [36],[61],[63], our evaluation is based on median IFA values. As674 illustrated in Figure 6a, LocVul achieves a median IFA equal to 0, suggesting that the675 Seq2Seq model manages to include a vulnerable line in the generated lines in most of676 the functions (i.e., for at least half of the functions in the dataset). In contrast, the677 Self-Attention mechanism achieves a higher median IFA equal to 2. This observation678 19 (a) Comparison of Initial False Alarm (IFA). (↘) Lower IFA - Better. (b) Comparison of Effort@20%Recall. (↘) Lower Effort@20%Recall - Better. (c) Comparison of Recall@1%LOC. (↗) Higher Recall@1%LOC - Better. Figure 6: Cost-effectiveness evaluation of LocVul compared to the baseline Self-Attention method for line-level vulnerability detection. indicates that LocVul improves the baseline approach in terms of the number of false679 alarms to inspect before finding a vulnerable line. In addition, as illustrated by the error680 bars representing half the Interquartile Range (IQR/2), LocVul demonstrates minimal or681 no variability, consistently achieving very low IFA values across all samples as opposed682 to Self-Attention, which has a larger spread.683 Furthermore, Figure 6b demonstrates that LocVul achieves Effort@20%Recall equal684 to 0.57% against Self-Attention’s 0.61%. The smaller the Effort@20%Recall, the less685 effort is required to detect vulnerabilities. Therefore, a code reviewer that uses LocVul686 can put less effort into identifying 20% of the truly vulnerable lines. In addition, Figure687 6c shows that LocVul’s Recall@1%LOC is 29.8% in contrast to Self-Attention’s 28.6%.688 In other words, a code reviewer is able to detect 1.2% more vulnerable lines by inspecting689 the top-1% recommended lines when using LocVul instead of the baseline Self-Attention690 approach.691 In addition, we perform the Wilcoxon Signed-Rank Test [86] to judge if the differ-692 ences in the cost-effectiveness metrics between the LocVul and Self-Attention approaches693 are statistically significant. Using the metrics computed for 10 repetitions of the experi-694 ments with different random seeds, the p-values of the Wilcoxon test are equal to 0.00098,695 0.02108, and 0.01368 for median IFA, Effort@20%Recall, and Recall@1%LOC metrics,696 respectively. Although the differences in the means of the cost-effectiveness metrics are697 not large, all the p-values are lower than the 0.05 threshold and, therefore, we can state698 that the differences in cost-effectiveness between the two approaches are statistically sig-699 nificant. Hence, considering IFA, Effort@20%Recall, and Recall@1%LOC scores, it is700 concluded that:701 702 The Seq2Seq-based LocVul approach offers a cost-effective solution for linelevel vulnerability detection, reducing the effort required to identify actual vulnerable lines compared to the baseline Self-Attention explainability method. 703 20 6. Discussion704 In this section, we discuss the findings of our experimental analysis and provide useful705 insights. First, we explore the benefit gained from the line replacement mechanism706 that we implement to address LLM hallucinations in VD. Subsequently, implications to707 researchers and practitioners are provided, and then, the threats to validity of our study708 are disclosed.709 6.1. Impact of the most similar line replacement mechanism710 In Section 3.3, we described a similarity checking mechanism based on cosine similar-711 ity between the embeddings of tokenized lines. This mechanism is applied when running712 the LocVul Detector in order to check whether the line generated by the model is one of713 the lines of the function being analyzed and, if not, to replace it with the corresponding714 line by calculating which line is the most similar. In this way, the proposed methodology715 manages to handle cases where the Seq2Seq model returns as vulnerable lines that are not716 in the function (i.e., LLM hallucinations in this problem). Table 2 presents the evaluation717 of the LocVul approach with and without employing the similarity check mechanism.718 Table 2: Comparison of results achieved by LocVul with and without the most similar line replacement mechanism. Accuracy Metrics Cost-Effectiveness Metrics Approach A@10 P@10 R@10 MRR@10 MAP@10 Median IFA Effort@20%Recall Recall@1%LOC With Replacement 82.8% 26.9% 79.0% 79.4% 79.2% 0 0.57% 29.8% Without Replacement 80.6% 25.3% 74.3% 76.9% 76.5% 0 0.70% 25.4% As shown in Table 2, the mechanism applied to replace the most similar lines provides719 a benefit to the entire approach, which is reflected in all evaluation metrics used. In terms720 of accuracy-related metrics, we can see that the proposed mechanism achieves a Top-10721 Accuracy, Precision, and Recall gain equal to 2.2%, 1.6%, and 4.7%, respectively. It also722 provides a gain of 2.5% and 2.7% in terms of MRR@10 and MAP@10, which means that723 LocVul with similar line replacement manages not only to detect more vulnerable lines,724 but also to place them higher in the list of the returned vulnerable lines compared to725 LocVul without similar line replacement. One can also observe an advantage of the hal-726 lucinations handling mechanism with regard to cost-effectiveness. Although median IFA727 is still zero, both Effort@20%Recall and Recall@1%LOC are deteriorated when remov-728 ing this mechanism. Specifically, the former increases by 0.13% and the latter decreases729 by 4.4%, leading to the conclusion that without similar line replacement, more effort is730 required by the code reviewer to find actual vulnerabilities using LocVul without being731 confused by hallucinations.732 In addition, to verify the benefit of the similar line replacement mechanism, we per-733 form the Wilcoxon Signed-Rank Test [86] on the evaluation metrics obtained from ten734 repetitions of the experiments with different random seeds. Specifically, we conduct the735 test on both accuracy and cost-effectiveness metrics, namely Top-10 Accuracy, Top-10736 Precision, Top-10 Recall, MRR@10, MAP@10, Effort@20%Recall, and Recall@1%LOC,737 excluding median IFA, as it is zero in both cases. The Wilcoxon test returns p-values738 equal to 0.00098 for all metrics. All the p-values are below the 0.05 threshold indicating739 that there is a statistically significant benefit gained from the similar line replacement740 mechanism in terms of both accuracy and cost-effectiveness in VD.741 21 6.2. Implications742 Having presented a thorough evaluation of the proposed method, the results demon-743 strate that the Seq2Seq approach for VD addresses important limitations of prior methods744 and outperforms traditional VD techniques that rely on XAI. Unlike the state-of-the-art745 Self-Attention-based approach, a Seq2Seq model, such as CodeT5, can directly identify746 vulnerable lines in source code without ranking tokens by importance or depending on747 error-prone function-level prediction models, which often capture spurious data correla-748 tions. The experiments carried out in the context of RQ1and RQ2demonstrate that749 fine-tuning a Seq2Seq model in detecting line-level vulnerabilities achieves both higher ac-750 curacy and better cost-effectiveness than approaches based on interpreting function-level751 predictions. These findings encourage stakeholders to focus on Seq2Seq-based solutions752 for VD rather than using XAI-based techniques by default.753 Furthermore, we recommend that researchers further elaborate the Seq2Seq-based754 approach, examining specific Seq2Seq training paradigms, in order to develop even more755 accurate models. For instance, they could extend our analysis by comparing techniques756 such as question-answering, machine translation, text generation, etc. In addition, we757 suggest that researchers validate the accuracy of this approach on different datasets758 and programming languages. Research on constructing new datasets with fine-grained759 vulnerability labels is recommended, as well.760 Moreover, we suggest researchers to enrich the VD-related literature by experiment-761 ing with various LLMs, either of the same or larger scale. They could repeat our analysis762 by fine-tuning models such as PLBART [87], GPT-4 [88], and Mistral [89] on the Seq2Seq763 approach to localize vulnerabilities within functions. An interesting research direction764 is also the exploration of innovative methods from the rapidly evolving field of artificial765 intelligence, such as Reinforcement Learning from Human Feedback (RLHF) [90] to in-766 crease the accuracy of the proposed VD approach, and Mixture of Experts (MoE) [91]767 to enhance the detection of vulnerabilities of different categories.768 In addition, the accuracy of LocVul combined with its cost-effectiveness and the769 efficiency in its execution is an important practical advantage. Software development770 workflows that require realor near-real-time applications can benefit from models such771 as CodeT5, which, when fine-tuned, can detect vulnerable lines in source code without772 any substantial delay. Specifically, although training LLMs is a time-consuming process,773 the perception time of LocVul (i.e., average time required to analyze one function during774 model execution) is only 213.83 milliseconds (ms). Therefore, LocVul can detect with775 sufficient accuracy where a vulnerability is located in the source code of a function in776 213.83 ms, without necessarily requiring additional effort to inspect the line-ranking list,777 as in the case of XAI-based solutions. Thus, we recommend that practitioners use such778 solutions as copilots (e.g., through their Integrated Development Environments) in their779 daily development activities to improve the overall security and productivity of the SDLC780 with little or no disruptions to their regular workflows.781 Furthermore, practitioners are suggested to use LocVul during software development782 and testing, and to compare it with existing static code analysis solutions, which are783 traditionally used for identifying potential vulnerabilities through source code scanning.784 In this way, practitioners could provide useful insights regarding the comparison of solu-785 tions like LocVul against static code analysis tools, which often demand substantial time786 to analyze large code bases [92],[93]. Although transfer learning approaches have been787 22 shown to outperform static code analyzers in terms of accuracy [36], a holistic evaluation788 considering the computational footprint of each approach would be useful.789 6.3. Threats to validity790 A remark about threats to the validity of our approach is needed. Threats to con-791 struct validity concern the selection of the metrics used in our evaluation scheme. In792 our experimental analysis, we use A@10, P@10, and R@10 to measure the accuracy of793 LocVul and Self-Attention approaches. We also use MRR@10 and MAP@10 measure-794 ments to evaluate the accuracy of the compared approaches in a rank-aware manner. In795 addition, the use of the IFA, Effort@20%Recall and Recall@1%LOC metrics contributes796 to a comprehensive assessment that takes into account cost-effectiveness. These met-797 rics are widely used in the related literature [10],[36],[61],[63], which effectively mitigates798 construct validity risks.799 The primary threat to internal validity relates to hyperparameter selection when800 fine-tuning our LocVul model. Although we have performed extensive hyperparameter801 tuning, it is not certain that we have tested all possible combinations of hyperparameter802 values. It can be difficult to discern the effects of specific model hyperparameters due to803 the correlation between them, which could lead to a lower than ideal model performance.804 To reduce this risk, we use a quite thorough hyperparameter tuning procedure based on805 the Grid-search method [94].806 Moreover, a threat to internal validity relates to the accuracy and correctness of our807 own approach. To mitigate this concern, we have not only meticulously reviewed our808 code, but also aim to make the code publicly available [43] to facilitate future replication809 and validation of our findings. In addition, a potential threat to internal validity also810 is the accurate implementation of the state-of-the-art Self-Attention approach that we811 use as a baseline mechanism. It is important to ensure that this approach is faithfully812 and correctly reproduced. To this end, we thoroughly inspected and consulted the code813 provided by the LineVul study [36] and, additionally, we include the code we developed814 for the Self-Attention approach in our replication package, along with the code for the815 LocVul approach.816 Finally, external validity is related to the generalizability of the LocVul approach. Our817 study is limited to one dataset, which contains open-source projects written in C/C++818 language, and thereby, the findings may not extend to projects in other languages or819 proprietary software. We selected Big-Vul [68], which is a widely-referenced dataset in820 the vulnerability-related literature, especially in studies interested to line-level detection821 [36],[57],[58]. It consists of 348 different projects containing code changes and vulner-822 abilities retrieved from GitHub and the CVE database, respectively. The size and the823 trustworthiness of Big-Vul mitigates the generalizability risk. Other line-level datasets824 including projects in different programming languages could be explored in the future.825 7. Conclusions and future work826 This study identified limitations in existing vulnerability detection (VD) techniques827 and proposed the LocVul methodology, which is based on a Sequence-to-Sequence ap-828 proach. Specifically, we proposed a two-step mechanism, which first, fine-tunes a Large829 Language Model (LLM) in predicting the vulnerable functions (i.e., a classification task),830 23 and second, fine-tunes another LLM in extracting the vulnerable lines of a given func-831 tion. The analysis found that LocVul achieved beyond state-of-the-art results, managing832 to clearly surpass explainability-based solutions both in accuracy and cost-effectiveness.833 Useful insights were also gained about the effective handling of the LLM hallucinations834 in VD and about the potential of tools such as LocVul to act as a co-pilot in software835 development.836 Suggestions for future work include the construction of a complete AI-driven VD837 pipeline that will identify vulnerable software components, localize the specific lines of the838 vulnerabilities, classify them to vulnerability categories, and assign them severity scores.839 We aim also at examining the accuracy of LocVul to other programming languages and840 different application domains.841 Declaration of competing interest The authors declare no conflict of interest. Acknowledgments Work reported in this paper has received funding from the European Union’s Horizon Europe Research and Innovation Program through the DOSS project, Grant Number 101120270. CRediT authors’ contribution statement Ilias Kalouptsoglou: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Writing - Original Draft, Visualization. Miltiadis Siavvas: Methodology, Validation, Investigation, Writing - Review & Editing, Supervision. Apostolos Ampatzoglou: Methodology, Writing - Review & Editing, Supervision, Project administration. Dionysios Kehagias: Writing - Review & Editing, Resources, Supervision, Project administration, Funding acquisition. Alexander Chatzigeorgiou: Methodology, Writing - Review & Editing, Supervision, Project administration. References [1] ISO/IEC, ISO/IEC 25010 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models, ISO/IEC, 2011. [2] B. Boehm, V. R. Basili, Software defect reduction top 10 list, Software engineering: Barry W. Boehm’s lifetime contributions to software development, management, and research 34 (2007) 75. [3] G. McGraw, Software security, Building security in (2006). [4] Published CVE Records, https://www.cve.org/about/Metrics. [5] ISO/IEC 27000:2018, https://www.iso.org/obp/ui/\#iso:std:iso-iec:27000: ed-5:v1:en (2023). 24 [82] M. Ancona, E. Ceolini, C. Öztireli, M. Gross, Towards better understanding of gradient-based attribution methods for deep neural networks, arXiv preprint arXiv:1711.06104 (2017). [83] A. Shrikumar, P. Greenside, A. Kundaje, Learning important features through propagating activation differences, in: International conference on machine learning, PMlR, 2017, pp. 3145–3153. [84] S. Lundberg, A unified approach to interpreting model predictions, arXiv preprint arXiv:1705.07874 (2017). [85] LineVul, LineVul Replication Package, https://github.com/awsm-research/ LineVul (2022). [86] F. Wilcoxon, Individual comparisons by ranking methods, in: Breakthroughs in statistics: Methodology and distribution, Springer, 1992, pp. 196–202. [87] W. U. Ahmad, S. Chakraborty, B. Ray, K.-W. Chang, Unified pre-training for program understanding and generation, arXiv preprint arXiv:2103.06333 (2021). [88] OpenAI, Gpt-4 technical report, https://cdn.openai.com/papers/gpt-4.pdf (2023). [89] F. Jiang, Identifying and mitigating vulnerabilities in llm-integrated applications, Master’s thesis, University of Washington (2024). [90] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in neural information processing systems 35 (2022) 27730–27744. [91] Z. Chen, Y. Deng, Y. Wu, Q. Gu, Y. Li, Towards understanding the mixture-ofexperts layer in deep learning, Advances in neural information processing systems 35 (2022) 23049–23062. [92] M. Siavvas, E. Gelenbe, D. Kehagias, D. Tzovaras, Static analysis-based approaches for secure software development, in: Security in Computer and Information Sciences: First International ISCIS Security Workshop 2018, Euro-CYBERSEC 2018, London, UK, February 26-27, 2018, Revised Selected Papers 1, Springer International Publishing, 2018, pp. 142–157. [93] B. Johnson, Y. Song, E. Murphy-Hill, R. Bowdidge, Why don’t software developers use static analysis tools to find bugs?, in: 2013 35th International Conference on Software Engineering (ICSE), IEEE, 2013, pp. 672–681. [94] J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl, Algorithms for hyper-parameter optimization, Advances in neural information processing systems 24 (2011). 31