SafeFetch: Practical Double-Fetch Protection with Kernel-Fetch Caching

This paper is included in the Proceedings of the 33rd USENIX Security Symposium. August 14–16, 2024 • Philadelphia, PA, USA 978-1-939133-44-1 Open access to the Proceedings of the 33rd USENIX Security Symposium is sponsored by USENIX. SafeFetch: Practical Double-Fetch Protection with Kernel-Fetch Caching Victor Duta, Mitchel Josephus Aloserij, and Cristiano Giuffrida, Vrije Universiteit Amsterdam https://www.usenix.org/conference/usenixsecurity24/presentation/duta SafeFetch: Practical Double-Fetch Protection with Kernel-Fetch Caching Victor Duta Mitchel Josephus Aloserij Cristiano Giuffrida Vrije Universiteit Amsterdam Abstract Double-fetch bugs (or vulnerabilities) stem from in-kernel system call execution fetching the same user data twice without proper data (re)sanitization, enabling TOCTTOU attacks and posing a major threat to operating systems security. Existing double-fetch protection systems rely on the MMU to trap on writes to syscall-accessed user pages and provide the kernel with a consistent snapshot of user memory. While this strategy can hinder attacks, it also introduces nontrivial runtime performance overhead due to the cost of trapping/remapping and the coarse (page-granular) write interposition mechanism. In this paper, we propose SafeFetch , a practical solution to protect the kernel from double-fetch bugs. The key intuition is that most system calls fetch small amounts of user data (if at all), hence caching this data in the kernel can be done at a small performance cost. To this end, SafeFetch creates per-syscall caches to persist fetched user data and replay them when they are fetched again within the same syscall. This strategy neutralizes all double-fetch bugs, while eliminating trapping/remapping overheads and relying on efficient byte-granular interposition. Our Linux prototype evaluation shows SafeFetch can provide comprehensive protection with low performance overheads (e.g., 4.4% geomean on LMBench), significantly outperforming state-of-the-art solutions. 1 Introduction The operating system (OS) kernel is the bedrock of modern systems. To provide service, the kernel includes a syscall interface, an explicit boundary between untrusted OS processes and the trusted kernel. Hence, it is crucial for the kernel to properly sanitize data that flows through this boundary (e.g., syscall arguments). Failure to do so may lead to (kernel) double-fetch bugs [15]. Such bugs occur when the kernel fetches (i.e., reads) the same/overlapping data from user space twice—a common kernel design pattern—without properly (re)sanitizing data on the second fetch. In essence, double-fetch bugs introduce a race condition, which attackers can exploit to mount timeof-check to time-of-use (TOCTTOU) attacks—changing user data between the two fetches. This is to bypass sanity checks and typically escalate privileges. Such bugs are both common (as they involve skipping seemingly “redundant” kernel checks [15]) and elusive (as they normally escape testing with production sanitizers [9]). Prior research [18,23,26,31] has mostly sought to detect and report several double-fetch bugs [2 – 7]. However, prior detection tools are imprecise and thus unsuitable to mitigate double-fetch bugs in production. More recently, Midas [15] proposed the first mitigation to offer protection (rather than detection) guarantees against double-fetch bugs. Midas relies on the MMU and copy-on-write mechanics to expose consistent snapshots of user pages to each syscall. To this end, Midas traps writes to each fetched user page, copies the page, and exposes the new (old) page to the writer (syscall). While this approach structurally prevents double-fetch exploitation, it also incurs nontrivial overhead due to the cost of trapping, remapping, and copying pages as well as operating at the coarse page granularity (causing overtrapping and overcopying due to false sharing [15]). Finally, due to the complex and costly operations Midas whitelists a number of syscalls (including the kernel-fetch heavy execve ), ultimately reducing protection coverage. In this paper, we propose SafeFetch , a practical protection system against kernel double-fetch bugs. The key idea is to move the core instrumentation from writes to kernel fetches, with the kernel maintaining per-syscall caches to serve kernel fetches. Indeed, our design has been inspired by Linux kernel developers seeking a practical double-fetch bug mitigation by “performing some kind of kernel-side caching of user space memory” [8]. SafeFetch ’s design prevents concurrent (attacker-controlled) writes from corrupting data exposed to kernel double fetches—which instead hit the previUSENIX Association 33rd USENIX Security Symposium 1207 ously populated kernel-fetch cache by construction. As we will show, the vast majority of syscalls only copy a few bytes from user space, hence caching kernel-fetch data per-syscall can be done in a simple and inexpensive way. In contrast to Midas’ write-side instrumentation strategy, SafeFetch ’s fetch-side strategy eliminates the need for costly MMU-based instrumentation (as writes run uninstrumented), false sharing and overcopying (as the cache operates at the byte rather than page granularity), and syscall-based whitelisting (as individual problematic kernel fetches can be whitelisted as needed). As a result, SafeFetch significantly improves both the performance and the security (protection coverage) of state-of-the-art solutions at a fraction of the complexity. To support our claims, we implemented SafeFetch on Linux and evaluated our prototype, along with a number of caching-centric optimizations, against a number of standard benchmarks. Our evaluation shows that SafeFetch provides comprehensive protection at a fraction of the overhead incurred by Midas, despite the higher protection coverage (i.e., a single kernel fetch vs. three major syscalls whitelisted). For instance, SafeFetch incurs geomean performance overheads consistently below 5% across standard kernel benchmarks (i.e., LMBench, OSBench, and Phoronix). On the same benchmarks, Midas reports much higher geomean overheads (i.e., as high as ≈ 15% on OSBench and ≈ 36% on LMBench). Moreover, Midas incurs a single-benchmark worst-case overhead of 279% (vs. 22% for SafeFetch). Contributions. We make the following contributions: • We investigate common kernel fetch patterns during syscall execution and use the resulting insights to design a per-syscall kernel-fetch cache. • We present SafeFetch , an implementation of our design to structurally mitigate double-fetch bugs in the Linux kernel. We show SafeFetch can be seamlessly integrated into existing kernel code paths, resulting in a practical implementation. • We evaluate SafeFetch on a number of standard benchmarks, confirming that it can comprehensively mitigate double-fetch bugs with low performance overheads (e.g., 4.4% geomean on LMBench). 2 Background 2.1 User/kernel Memory Isolation Modern operating systems rely on virtual memory support to enforce user/kernel memory isolation, that is preventing user (kernel) execution from accessing kernel (user) memory. This is typically done by using a joint virtual memory address space—where both user and kernel USERSPACE @data Thread 1 Thread 2 syscall_entry FETCH @data VALIDATE @data FETCH @data USE @data UPDATE @data TIME Figure 1: Workflow of a double-fetch exploit. memory mappings coexist during user/kernel execution— and features offered by modern memory management units (MMUs) to enforce isolation. Specifically, on x86 platforms, the kernel can set (unset) the User/Supervisor bit in the Page Table Entries (PTEs) for user (kernel) memory mappings. This prevents user execution from accessing kernel memory. It also prevents the reverse (i.e., kernel execution accessing user memory) assuming Supervisor Mode Access/Execution Prevention features (SMAP and SMEP, respectively) are enabled. While important for memory isolation, SMAP complicates the implementation of common operations such as kernel fetches, that is user-to-kernel data transfers often issued by the kernel as part of syscall handling (e.g., copying a message from user memory to be sent over the network). To ease their implementation, modern operating systems typically support special kernel transfer functions in order to safely transfer data between user and kernel. For example, the Linux kernel offers two userto-kernel (i.e., copy_from_user and get_user ) and two kernel-to-user (i.e., copy_to_user and put_user ) transfer functions, which temporarily disable SMAP and copy data from/to user memory (respectively). 2.2 Double-fetch Bugs A kernel double fetch occurs in presence of kernel fetches transferring the same user data twice, that is with multiple user-to-kernel transfer function invocations for the same (or overlapping) user data on Linux. This pattern is normally benign and used to simplify or optimize common types of (e.g., deep or variable-length [26]) userto-kernel data transfers. However, if the kernel assumes the data to be invariant and only validates data on the first fetch, the second fetch originates a double-fetch bug. Such bugs are particularly insidious as they introduce a race condition that may never cause any harm during normal execution. However, an attacker can exploit such 1208 33rd USENIX Security Symposium USENIX Association Thread syscall_entry FETCH @data FETCH @data syscall_exit TIME SafeFetch per-thread caches Userspace search @data (cache miss) fetch @data from user store @data return @data search @data (cache hit) return @data invalidate cache Figure 2: SafeFetch hindering a double-fetch exploit. bugs by racing against kernel execution from another user thread and corrupting the (unsanitized) data exposed to the second fetch. Such time-of-check to time-of-use (TOCTTOU) attack can bypass sanity checks and often kickstart a privilege escalation exploit [26]. Figure 1depicts the workflow of a typical double-fetch exploit. In response to Thread 1 executing a syscall, the kernel first fetches and validates some user @data . Shortly after, another attacker-controlled Thread 2 concurrently modifies the @data in user space. In Thread 1, the kernel then proceeds to fetch @data again without (re)validation. This allows the attacker to bypass validation checks and mount a TOCTTOU attack. Prior work has largely focused on detecting such bugs with reasonable accuracy [18,23,26,31]. In this paper, we instead focus on protecting the kernel from zero-day double-fetch bugs, while proposing a much simpler and more efficient design than the state-of-the-art protection system [15]. 3 Threat Model We assume a typical local exploitation threat model, with an unprivileged user-space attacker seeking to exploit a kernel double-fetch bug. Other classes of vulnerabilities are out of scope, e.g., addressed by orthogonal mitigations. The attacker ultimately aims to mount a TOCTTOU attack for a variety of different purposes, e.g., privilege escalation, info leak, denial of service, etc. 4 Overview To hinder exploitation of kernel double-fetch bugs, SafeFetch guarantees that, during the lifetime of each syscall, kernel fetches to the same user data will return Cache Backend Cache Frontend SafeFetch Syscall Cache syscalltransfer function call search range range query miss Custom Allocator sanitized range provision Figure 3: SafeFetch’s high-level architecture. the same value. To this end, SafeFetch caches data read by kernel fetches at the per-thread and per-syscall granularity, as illustrated in Figure 2. As shown in the figure, when a syscall fetches some user @data for the first time, SafeFetch proxies the fetch to user memory and stores the fetched data in a per-thread in-kernel cache. When the syscall fetches the same @data again, SafeFetch retrieves the data from the cache. As such, double fetches are always consistently served with the initial version of the data regardless of any concurrent updates to user memory. At the end of the syscall lifecycle, the cache is invalidated to implement per-syscall caching semantics. Internally, SafeFetch includes two core components, as shown in Figure 3. The Cache Frontend intercepts each kernel fetch, i.e., (user-to-kernel) transfer function invocation, and returns a sanitized range (i.e., a contiguous, immutable user memory block) in output. To this end, the frontend queries the current syscall cache for the range. In case of a hit, a (sub)range is served directly from the cache. In case of a miss, the frontend notifies the Cache Backend . The latter provisions the cache to store the missing (sub)range (fetched from user memory) by means of a custom allocator. Challenges. While SafeFetch ’s design is conceptually simple, there are several challenges involved in its realization. First, the need for an in-kernel cache may lead to a nontrivial TCB impact, affecting security. We will later show it is feasible to efficiently implement our design with small TCB impact. Second, the need to interpose on all fetch operations may lead to protection coverage (and thus security) issues. We will later show it is feasible to produce a (nearly) full-coverage implementation on modern operating systems such as Linux, faring even better than the state of the art (Midas). Third, our fetch-side instrumentation strategy may end up copying USENIX Association 33rd USENIX Security Symposium 1209 more data than Midas’ write-side strategy in case user data is never changed during syscall execution. We will later show such cost is marginal compared to that of MMU-based instrumentation, resulting in consistently better performance. Finally, our design require instrumenting the kernel’s fast path (i.e., transfer functions). As such, its instrumentation and data structures need to be carefully designed to efficiently support typical kernel fetch patterns. We will show it is possible to capture a variety of different fetch patterns with relatively simple data structures. In the next sections, we first analyze the patterns relevant to our design. Then, we use the insights gathered from our analysis to detail our design. 5 Profiling Kernel Fetches While our syscall caches are superficially similar to other kernel caches since they may support similar range queries, our design requirements are fairly unique. For instance, the VMA cache is a classic example of a kernel cache supporting range queries, however, its lookup patterns are wholly different from ours (lookups on localityfriendly memory management operations vs. lookups on kernel fetches) and so are its scope (process vs. syscall) and data storage requirements (fixedvs. variable-sized data). As such, to make optimal design decisions, we need to learn more about typical patterns for kernel fetches, including their frequency, data transfer size, etc. To this end, we developed a simple profiler to gather kernel-fetch statistics. Specifically, our profiler interposes on syscall execution and records the following statistics: the total number of ranges a syscall transfers from user space, the average size of ranges transferred by a syscall, and the total amount of data a syscall fetches from user space. Moreover, for each process executed during profiling, we also gather the number of syscalls that transfer data from user space. We use various benchmarks (e.g., LMBench, OSBench) and popular user applications (e.g., Nginx, Apache) to generate a workload to sample syscall execution. In total, our workload generated around 317 million syscall samples, exercising 165 individual syscalls ( ≈ 52% of all defined syscalls for Linux x86_64 ). Our main profiling results are depicted in Figures 4,5,6,7,8. We elaborate on the results in the next sections, using the gathered insights to motivate our design. 6 Cache Frontend To provide the kernel with a consistent view of user memory, the cache frontend interposes on all user-tokernel transfer function invocations requesting a specific user range. In response, the frontend queries the syscall cache for the range by means of the user (virtual) address and the length of the range. After the query completes, the frontend performs a query resolution step, fetching parts of the range from user space into the cache if needed (i.e., if not cached) and then forwarding a sanitized range to the original transfer function. The frontend considers a user range A sanitized if and only if: for any sub-range B of contiguous user addresses, such that B⊆A , and B was previously fetched during the execution of the syscall (i.e., via a previous transfer function) then B must consist of the same bytes as when it was first fetched. When querying the cache, the frontend uses a predetermined search policy to locate the range in the cache. The search policy is subject to the (meta)data structure used to bookkeep the user ranges in the cache. 6.1 Efficient Range Queries Given a user start and end address (i.e., a range) SafeFetch needs to find all cached chunks that overlap with this input range. Since a range may only partially overlap with an existing range (or multiple cached ranges), we are interested in finding the optimal data structure that can service this operation efficiently. For this purpose, other kernel subsystems use either linked lists (for small caches) or red-black trees (for large caches). The Virtual Memory Area (VMA) cache is case in point, generally serving address range queries via a per-process red-black tree. However, the VMA cache’s fast path uses a linked list for the few recently used VMAs. We experimented with both types of data structures in the context of SafeFetch . In both cases, a node contains metadata recording the start/end address of the range and a reference to the cached data. Clearly, in the case of many cached ranges, we expect linked-list-based queries to perform poorly, with a worst-case search complexity of O(n). In the same vein, we expect red-black trees to be more efficient, with a worst-case search search complexity of O(log(n)) due to constant-time rebalancing. Indeed, we experimentally verified that after around 100 cached ranges, the average search time of a linked list is slowed down by a factor of two compared to a red-black tree. However, when the cache contains only a few elements we observed the linked list significantly outperforming a red-black tree. On top of more lightweight search logic, a small linked list has another important performance edge over a small red-black tree on the insertion path (i.e., when adding a new user range into the cache). Indeed, due to rebalancing, insertions in the red-black tree are always around 10 orders of magnitude slower than in a linked list. Since according to our profiling results, double fetches are rare (occurring once every 1,277 sampled syscalls), insertions are frequent and are thus important to consider for performance. For more detailed double fetch statistics from our profiling results, 1210 33rd USENIX Security Symposium USENIX Association 0 fetches 1 fetch [2-10) fetches [10-30) fetches [30-4099) fetches 0% 50% % of syscall samples 63.93% 28.69% 7.29% 0.03% 0.06% Figure 4: Percentage of syscall samples vs. number of ranges they fetch. we refer the interested reader to Appendix A. Selecting the ideal data structure. To select the ideal candidate between our two data structures, we turn to our profiling results. Figure 4shows the percentage of syscall samples that fetch N ranges from user memory across all the profiled benchmarks. As shown in the figure, most samples ( ≈ 64%) do not fetch user ranges at all, while the vast majority of those which do, fetch at most one range ( ≈ 28.7% of samples). Even so, a nontrivial number of samples ( ≈ 7.32%) fetch at most 30 ranges, while only ≈ 0.06% samples fetch over 30 ranges. As our results suggest, a (small) linked list seems the ideal candidate to support range queries for the vast majority of syscall samples. However, we also observed samples fetching as many as ≈ 4,000 ranges. In those cases, the linked list has very poor performance and the red-black tree is a vastly better option. To optimize for all possible syscall scenarios, we ultimately opted for an adaptive search policy. In other words, the search policy uses a linked list by default until the number of ranges in the cache reaches a predetermined threshold. When the threshold is exceeded, SafeFetch switches to a red-black tree implementation. We detail how we experimentally selected a good threshold in Section 9.3. To convert the linked list into a redblack tree, SafeFetch uses an efficient algorithm that constructs a balanced red-black tree from an ordered linked list. This is done by first copying the pointers to ranges in a vector and then iterating over the vector in a binary breath-first search fashion. To keep the algorithm efficient, we need to initiate the conversion when the list contains a number of ranges of the form 2N−1. 6.2 Query Resolution When processing the result of the query, the Cache Frontend makes different decissions depending on whether it found a cached user range colliding with the queried range (cache hit) or not (cache miss). In case of a miss, no subrange from the queried range was previously fetched from user space. As a resolution, the frontend fetches the range from user space and instructs the backend to allocate storage to insert the range into the cache. It then also indexes the new range by inserting new metadata into the linked list or red-black tree with with O(1) complexity—by piggybacking on the result of the previous query. Cache hits can either be perfect or partial. A perfect hit means that the frontend found a cached range which contains the entire range it queried for. In this case, the frontend forwards the sane range from the cache without performing any fetch from user space. A cache hit is partial if the frontend finds a cached range that collides with the range it queried for, but does not contain the entire range. In this case, the queried range may collide with multiple cached ranges previously fetched from user space and the frontend needs to first execute a range defragmentation step to determine the sanitized range. Let B be the queried range and let M={Ai|Ai∈cache ∧ AiTB6=/ 0} containing all cached ranges that collide with B . Defragmentation involves computing a new range C such that C=∪n i=1Ai∪(B\∪n i=1Ai) . In other words, the defragmented range C contains all bytes from the cached ranges colliding with B , while the sub-ranges of bytes that are not cached yet are fetched from user space. The frontend instructs the backend to replace all colliding ranges in the cache with the newly defragmented range C , after which it can service the sanitized range from C . Defragmentation piggybacks on the result of the previous search, because all colliding ranges can be found by using the previous search’s iterator. To support this, insertion in the linked list and red-black tree preserves the virtual address ordering of cached ranges. 7 Cache Backend The Cache Backend is responsible with managing the backing memory for the syscall cache. Specifically, its main goals are to efficiently manage the lifecycle of ranges in the cache and exploit CPU cache locality as much as possible. The latter can be achieved by stacking together ranges in memory for data locality and thus better CPU cache utilization, which speeds up range queries. The former involves enforcing policies for range allocation/deallocation and storage provisioning/relinquishing techniques to keep cache operations optimal. To achieve its goals, the backend maintains one cache for each in-transit syscall (at the per-thread granularity) and adheres to a cache organization specifically tailored to maximize data locality when performing range queries through the cache. Additionally, the backend enforces a series of range lifecycle management policies to efficiently oversee cache lifespan. To support this overall strategy, the backend relies on a custom memory allocator. USENIX Association 33rd USENIX Security Symposium 1211 2022242628210 212 214 Average amount of bytes per fetch 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% % of syscall samples Figure 5: Percentage of syscall samples vs. average size of data they fetch. 7.1 Custom Allocator A naive allocation policy would be to create an object in the cache every time a syscall fetches a new range, by using an of-the-shelf kernel allocator (e.g., slab) to accommodate range data and metadata. However, standard kernel allocators do not give control over where objects are placed in (virtual) memory, so we cannot assure locality to improve the performance of range queries. Moreover, for an off-the-shelf allocator the allocation and deallocation logic might cause non-trivial overhead when syscalls transfer many ranges from user space, which happens in practice (see Figure 4). A better approach is to service kernel memory in larger chunks to fit multiple ranges, i.e., using a region-based allocation scheme [13]. In such a scheme, a region consists of one or more buffers (contiguous memory blocks) with the same data lifetime. In a region, memory objects are allocated one after another in memory (in the current buffer) and get deallocated all at once (by flushing all the buffers), once the lifetime of all the objects in the region ends. As such, region-based allocation allows us to pack ranges together in memory to improve locality. Moreover, it reduces the number of calls to the underlying allocator when allocating ranges. On the fast path, an allocation involves only bumping a pointer to the next slot in the current buffer. Finally, it can efficiently deallocate all the allocated objects in one blow. SafeFetch uses a custom region-based allocator, which services kernel memory at the granularity of a region, every time the backend requests more storage to hold ranges. To understand how well SafeFetch can benefit from region allocation’s locality-friendly design, we turn again to our profiling results. Figure 5shows the percentage of syscall samples that fetch an average size S of user data across all the profiled benchmarks. As shown in the figure, the majority (i.e., 65%) of syscall samples fetch ranges that are on average less than 64 bytes, confirming a high degree of data locality in a re20232629212 215 Total amount of bytes copied from user. 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% % of syscall samples Figure 6: Percentage of syscall samples vs. total amount of data they fetch. gion in the common case. SafeFetch also benefits from the fast deallocation path of region-based allocation, efficiently deallocating all the ranges in the region when the syscall terminates. As an optimization, SafeFetch does not discard the buffers once the region is deallocated, but adds them to a pool for (fast) reuse in new regions created by future syscalls. The next question is the buffer size one should use. As Figure 5suggests, small allocation requests are common suggesting small buffers are desirable. Moreover, Figure 6 shows the percentage of syscall samples that fetch a total amount of B bytes of user data across all the profiled benchmarks. As shown in the figure, although a moderate portion of syscalls transfer much data (even above 8 pages), the vast majority transfer far less than a page of data. As a result, SafeFetch uses a single 1-page buffer by default in each new region (syscall) and elastically adds buffers as needed in the edge cases—many fetches per syscall or fetches transferring over 4 KB of data. 7.2 Dual Region Design A naive approach would be to store the user memory ranges and the metadata necessary for range queries as a standalone object in one single per-syscall region. However, this approach would lead to metadata fragmentation and poor locality because metadata would be intermixed with data bytes. While most fetched ranges are small, we saw that syscalls can transfer larger ranges as well (Figure 5). To maximize data locality for range querying, the Cache Backend partitions the syscall cache in two separate regions: a data region stores all byte ranges copied from user space while a metadata region stores the bear bone necessities to perform queries over ranges. Consequently, for each user range, the backend maintains two objects: a) a data object storing the range of bytes copied from user space and b) a metadata object storing the properties of the range used when querying (e.g., user virtual address, length, pointer to the data object, and 1212 33rd USENIX Security Symposium USENIX Association 1 syscall [2-10) syscalls [10-50) syscalls 50 or more 0% 50% % of processes 0.4% 85.2% 5.6% 8.9% Figure 7: Percentage of processes vs. number of syscalls fetching user data. a field used to link into a red-black tree or linked-list). The size of metadata objects is fixed and small, allowing one to provision metadata regions with buffers smaller than a page. Nonetheless, we chose to serve 1page buffers to metadata regions as well because this leads to better CPU cache coloring (and utilization) [19]. Despite the same default buffer size, the two regions provision their buffers from separate memory pools (i.e., slab caches) to further improve locality. 7.3 Lifecycle Management Range allocation policy. A range is allocated in the cache every time the frontend encounters a cache miss while performing a range query. Our profiling data (e.g., Figure 4) suggests that most fetches are not double fetches, hence we expect frequent range allocations in the cache. Our custom allocator helps reduce the pressure on the (less efficient) underlying allocator, because many ranges can be allotted from the same region buffer. Allocating ranges entails creating metadata and data objects in the appropriate regions. To this end, the backend uses a region accounting structure, which keeps track of all buffers allotted to the referent region. To speedup object creation, the region accounting structure stores a pointer to a region head buffer and favors servicing allocations from this buffer. As region heads get depleted at some point, new buffers are added to the region when appropriate and region heads get updated. If an object cannot be created from the region head, the Cache Backend goes on the slow path iterating over all buffers allotted to the region until it finds one with enough space to service the request. As shown in Figure 6, some syscalls can transfer large amounts of user data, thus it is possible that a region can hold many depleted buffers. While this is more likely for data regions, it can also occur for metadata regions when syscalls execute many fetches. To optimize the slow path, the region accounting structure keeps a freelist containing only the buffers that are not yet depleted and can still service allocations. Lastly, if an object cannot be serviced on 0 bytes [1,64) bytes [64, 256) bytes [256-1024) bytes [1024-4096) bytes [1-4) pages [4,16) pages above 16 pages write pwrite64 writev sendto execve 0.01 0.17 0.40 0.10 0.13 0.01 0.18 0.01 0.03 0.56 0.21 0.10 0.07 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.80 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.57 0.39 0.03 0.00 0.0 0.2 0.4 0.6 0.8 ratio Figure 8: Heatmap showing the total amount of user data fetched by fetch-heavy syscalls. the slow path, then the backend leverages the custom allocator to create a new buffer in the referent region. Cache invalidation policy. When a system call terminates, all ranges currently held in the cache can be deallocated. As a result, one could relinquish all storage held by cached ranges on syscall exit. This strategy reduces the memory footprint and also yields a simple deallocation path flushing all the buffers held by metadata and data regions. Moreover, as discussed, syscalls do not fetch many ranges, thus on average we need not free many buffers. However, as shown in Figure 7, our profiling data shows that 99.6% of the processes that incur fetches do so across at least two syscalls. In other words, if a process issues a syscall that fetches user data, it is likely to issue other syscalls that do the same. Given our profiling results, it may be tempting to preserve all the allocated region buffers across syscalls. However, while this strategy minimizes the number of calls to the underlying allocator (improving performance), it may also significantly increase the steady-state memory footprint. That is particularly the case for threads issuing syscalls that fetch a lot of user data (even more than 8 pages, as shown in Figure 6). Given these observations, our cache invalidation policy is to preserve the head buffers, for both metadata and data regions, across the syscalls of a given thread but release all the other buffers held by each region, on syscall exit. Additionally, on each syscall exit, we invalidate the bookkeeping structure used by our frontend when performing range queries. Finally, on thread exit we release the residual head buffers. Cache initialization policy. Given that our two head buffers persist across syscalls of the same thread, the next question is when to allocate such buffers. An option would be to simply allocate the head buffers at thread (i.e., task_struct ) creation time. However, as shown in Figure 4, syscalls rarely fetch user data, e.g., ≈ 64% of the syscalls do not issue any fetches. Hence, eager head USENIX Association 33rd USENIX Security Symposium 1213 buffer allocation at thread creation time may lead to unnecessary memory overhead. As a result, the backend allocates the two head buffers (and the two regions) lazily, on the first user fetch that a thread incurs. Zero-copy optimization. Given that most syscalls fetch little data, storing data into the cache is generally inexpensive. However, some syscalls are fetch-heavy and copy large amounts of data. In such scenarios, copying data into the cache incurs a nontrivial cost (as shown later in our evaluation), so it is desirable to eliminate as many such extra copies as possible. To investigate optimization avenues, we isolated all syscalls in our profiling results that copy more than a page of data from user space. Figure 8presents our results in a heatmap. As shown in the figure, many of these syscalls (with the exception of exec) are I/O bound. To avoid resource contention, I/O bound syscalls rely on the kernel iov_iterator functionality to copy dozens of large chunks of user data into kernel staging buffers. Such buffers (and their copied data) persist unchanged until the hardware (e.g., network card, hard disk) becomes available to complete the I/O transfer. To optimize out expensive copies to the cache, we use the kernel staging buffer itself as a data cache for all the iov_iterator copies fetching one or more pages. To delay deallocation of such staging buffers until syscall exit, we increase the reference count on the underlying pages. When enabling this zero-copy optimization ( SafeFetch default), our cache invalidation policy also releases the staging buffers on syscall exit. 8 Implementation We implemented a SafeFetch prototype for Linux kernel v5.11 (matching Midas’ version for a fair comparison). The Cache Frontend adds the logic for cache lookups and fetch sanitization by instrumenting all kernel transfer function variants (i.e., raw_copy_from_user and all the get_user macro variants). To manage the cache lifecycle for in-transit syscalls, the Cache Backend stores the accounting structures for (metadata and data) caches as structures in a thread’s task_struct . Similarly, the bookkeeping structure used (for cache lookups) by the frontend is also referenced by the thread’s task_struct . The custom region allocator uses two global slab caches ( kmem_caches ) to service region buffers efficiently to in-transit syscalls. We implemented cache invalidation by instrumenting the epilogue of the do_syscall_64 kernel function and the do_exit function. Additionally, we instrumented all clone variants to initialize the SafeFetch logic in newly created threads. Finally, we implemented the zero-copy optimization by instrumenting the copyin kernel function used for iov_iterator-based user-to-kernel transfers. Maintainability. While our design and implementation are optimized based on our syscall profiling data, we efficiently support a variety of syscall patterns. As such, even assuming applications change some of their syscall patterns in the future, we expect only minor (if any) modifications to our implementation. To facilitate the process, SafeFetch already supports runtime changes for all its core parameters, similar to other workload-sensitive kernel features (e.g., KSM, THP, etc.). In other words, with its ability to support many syscall patterns, its small codebase, and compliance to standard kernel design patterns (e.g., use of standard kernel data structures, standard hooking points, etc.), we expect SafeFetch to be highly maintainable moving forward. 9 Evaluation In this section, we experimentally evaluate the security and performance of SafeFetch. 9.1 Security CVE analysis. As SafeFetch hinders double-fetch bugs by construction, similar to Midas [15], we verify it can correctly mitigate a known double-fetch vulnerability. CVE-2016-6516 is a known double-fetch vulnerability introduced in Linux kernel version 4.5. This vulnerability can be triggered via an ioctl syscall in combination with the FIDEDUPERANGE flag. This particular syscall attempts to deduplicate the memory pages between the source and its respective destination files. The source and destinations are supplied as parameters in a file_dedupe_range structure which the syscall copies from user space. However, before the syscall can fetch the structure it first needs to compute its size by using a count variable located inside the user space structure. After acquiring the count from user space, and performing sanity checks, the syscall fetches the entire structure into kernel memory overwriting the count field in the process, as can be seen in Listing 1. A double fetch vulnerability can emerge if between the first fetch (line 2 ) and second fetch (line 17 ) the count field is modified in user space. As a result, vfs_dedupe_file_range would operate on a malicious count value. This vulnerability was patched within Linux kernel v4.7 by copying back the old value into the data structure (line 23) after the second fetch. To evaluate whether SafeFetch can defend against this particular double fetch we modified Linux kernel version 5.11 by adding an additional check that verifies if a double fetch has occurred (lines 19-20 ) prior to the proposed fix. In order to reproduce the bug 1214 33rd USENIX Security Symposium USENIX Association Table 4: SafeFetch-induced throughput degradation. Benchmark SafeFetch /wo zero-copy /w zero-copy (%) (%) AF_UNIX 39.1% 8.4% Pipe 16.1% 5.1% Nginx 5.9% 1.3% Apache 9.3% 1.8% without zero-copy optimizations, SafeFetch copies large chunks of user data to the data cache, incurring significant overhead. For instance, avoiding unnecessary copying reduces the throughput degradation to 8.4% and 5.1% on the AF_UNIX and pipe benchmarks, respectively. Copying large chunks of user data impacts performance of real-world applications as well: Nginx and Apache have their throughput reduced by 5.9% and 9.3% compared to the baseline (respectively). Enabling the zero-copy optimization reduces throughput degradation for both web servers to below 2%. Our results confirms the zero-copy optimization is key to good I/O benchmark performance. 10 Related Work While double-fetch or TOCTTOU (Time-of-Check-toTime-of-Use) vulnerabilities affect different interfaces and components (e.g., enclaves [17,25], sandboxes [22], compilers [29], and compartments [14]), we focus here on closely related research on operating system kernels. Serna [24] was the first who coined the name “doublefetch vulnerability” to describe an instance in the Windows kernel. Since then, research has focused on finding double-fetch bugs (through static and dynamic program analysis) or even mitigating these issues. Static analysis. On the static analysis front, Wang et al. [26] leverages pattern matching analysis on source code to find double-fetch bugs in the Linux kernel. DFTinker [20] extends such pattern-based approach to increase double-fetch coverage and reduce false positives. While pattern-matching techniques proved effective in uncovering new double-fetch bugs, they still produce a high rate of false positives and negatives and are fundamentally limited to detecting only specific bug patterns. DEADLINE [31] and DFTracker [27] improve static detection of double fetches by means of compiler-level symbolic execution. While these approaches can be applied more broadly (e.g., to detect compiler-induced double fetches [28,30]) and are generally better suited for vetting false positives, symbolic execution introduces other limitations (e.g., path explosion) and does not completely remove false reporting (e.g., due to imprecise memory modeling or incomplete code coverage). DEADLINE, for example, does not detect double fetches in inline assembly, which is widely used in the kernel. Dynamic analysis. On the dynamic analysis front, Jurczyk and Coldwind [18] propose Bochspwn, which instruments memory access callbacks in an x86 emulator to trace double-fetch patterns. Coupled with fuzzing, their technique found a series of exploitable double fetches in the Windows kernel. However, Bochspwn incurs high overheads because its tracing instrumentation affects the emulator’s fast path. Wilhelm [28] uses a similar approach to Bochspwn and found double-fetch vulnerabilities in the Xen hypervisor, one of which was introduced through compiler optimizations. Schwarz et al. [23] use a fuzzer in conjunction with concurrent user memory accesses to detect double fetches through a cache covert channel. However, such a solution is reliant on hardware features (e.g., caches), which vary across microarchitectures and are subject to noise. While dynamic techniques are often more precise than static approaches, they are fundamentally subject to code coverage and limited to finding double-fetch bugs only on executed code paths. Mitigations. Midas [15] is the first proposed mitigation to protect the operating system kernel against doublefetch vulnerabilities. Midas relies on MMU-enabled Copyon-Write semantics to create snapshots of user pages accessed by the kernel during syscall execution. As a result, Midas incurs nontrivial runtime performance overhead due to the cost of trapping/remapping and the coarse (page-granular) write interposition mechanism. Additionally, Midas whitelists some syscalls, most notably the fetch-heavy exec syscall. In contrast, SafeFetch eliminates the need for MMU-based instrumentation and offers comprehensive protection at a fraction of Midas’ cost. Moreover, our solution is simple and seamlessly integrates with existing kernel code paths. Indeed, SafeFetch ’s high-level design is inspired by the “kernel-side caching of user space memory” on the wishlist of Linux kernel developers to address double-fetch vulnerabilities [8]. 11 Conclusion We presented SafeFetch , a practical double-fetch bug protection system which caches kernel fetches at the syscall granularity and serves subsequent fetches of the same data from the cache, thus ensuring that user data never changes across fetches. We showed that SafeFetch offers comprehensive protection at a fraction of the cost of state-of-the-art solutions such as Midas with marginal memory overheads and geometric performance overheads consistently below 5% across various OS benchmarks (e.g., 4.4% on LMBench and 1.3% on OSBench) and real-world workloads (e.g., 1.2% on Phoronix). USENIX Association 33rd USENIX Security Symposium 1221 Availability To encourage adoption, we have open sourced SafeFetch at https://github.com/vusec/safefetch . We are also actively engaging Linux kernel developers to seek mainline inclusion. Acknowledgments We would like to thank the anonymous reviewers for their feedback. This work was supported by Intel Corporation through the “Allocamelus” project, by NWO through project “INTERSECT” and “Theseus”, and by the European Union’s Horizon Europe programme under grant agreement No. 101120962 (“Rescale”). References [1] Best practices for adapting phoronix test suite to benchmark linux performance. https://blogs.oracle.com/linux/post/bestpractices-for-adapting-phoronix-testsuite-to-benchmark-linux-performance. [2] Cve-2013-1332. https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2013-1332. [3] Cve-2015-8550. https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2015-8550. [4] Cve-2016-10433. https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2016-10433. [5] Cve-2016-10435. https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2016-10435. [6] Cve-2016-10439. https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2016-10439. [7] Cve-2016-8438. https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2016-8438. [8] Detect and avoid ToCToU double-fetch / doubleread from userspace. https://github.com/KSPP/ linux/issues/95. [9] KASAN. https://github.com/google/kasan/ wiki. [10] Ni linux device drivers fails to install daqmx. https://knowledge. ni.com/KnowledgeArticleDetails?id= kA03q000000wzMJCAY. [11] OSBench Authors. OSBench. https://https:// github.com/mbitsnbites/osbench. [12] Phoronix Authors. Phoronix. https://www. phoronix-test-suite.com. [13] Emery D Berger, Benjamin G Zorn, and Kathryn S McKinley. Reconsidering custom memory allocation. In OOPSLA, 2002. [14] Atri Bhattacharyya, Florian Hofhammer, Yuanlong Li, Siddharth Gupta, Andres Sanchez, Babak Falsafi, and Mathias Payer. Securecells: A secure compartmentalized architecture. In IEEE S&P, 2023. [15] Atri Bhattacharyya, Uros Tesic, and Mathias Payer. Midas: Systematic kernel TOCTTOU protection. In USENIX Security, 2022. [16] Aaron B Brown and Margo I Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of netbsd on the intel x86 architecture. In SIGMETRICS, 1997. [17] Felix Dreissig, Jonas Röckl, and Tilo Müller. Compiler-aided development of trusted enclaves with rust. In ARES, 2022. [18] Mateusz Jurczyk and Gynvael Coldwind. Identifying and exploiting windows kernel race conditions via memory access patterns. 2013. [19] Hsien-Hsin S Lee and Gary S Tyson. Region-based caching: an energy-delay efficient memory architecture for embedded processors. In CASES, 2000. [20] Yingqi Luo, Pengfei Wang, Xu Zhou, and Kai Lu. Dftinker: Detecting and fixing double-fetch bugs in an automated way. In WASA, 2018. [21] Larry W McVoy, Carl Staelin, et al. lmbench: Portable tools for performance analysis. In USENIX ATC, 1996. [22] Shravan Narayan, Craig Disselkoen, Tal Garfinkel, Nathan Froyd, Eric Rahm, Sorin Lerner, Hovav Shacham, and Deian Stefan. Retrofitting fine grain isolation in the firefox renderer. In USENIX Security, 2020. [23] Michael Schwarz, Daniel Gruss, Moritz Lipp, Clémentine Maurice, Thomas Schuster, Anders Fogh, and Stefan Mangard. Automated detection, exploitation, and elimination of double-fetch bugs using modern cpu features. In AsiaCCS, 2018. [24] Fermin J. Serna. Ms08-061 : The case of the kernel mode double-fetch. https://msrc.microsoft. com/blog/2008/10/ms08-061-the-case-ofthe-kernel-mode-double-fetch/, 2008. 1222 33rd USENIX Security Symposium USENIX Association [25] Jo Van Bulck, David Oswald, Eduard Marin, Abdulla Aldoseri, Flavio Garcia, and Frank Piessens. A tale of two worlds: Assessing the vulnerability of enclave shielding runtimes. In CCS, 2019. [26] Pengfei Wang, Jens Krinke, Kai Lu, Gen Li, and Steve Dodier-Lazaro. How double-fetch situations turn into double-fetch vulnerabilities: A study of double fetches in the linux kernel. In USENIX Security, 2017. [27] Pengfei Wang, Kai Lu, Gen Li, and Xu Zhou. Dftracker: detecting double-fetch bugs by multi-taint parallel tracking. Frontiers of Computer Science, 13, 2019. [28] Felix Wilhelm. Xenpwn: Breaking paravirtualized devices. Black Hat USA, 2016. [29] Jianhao Xu, Luca Di Bartolomeo, Flavio Toffalini, Bing Mao, and Mathias Payer. Warpattack: Bypassing cfi through compiler-introduced double-fetches. In IEEE S&P, 2023. [30] Jianhao Xu, Kangjie Lu, Zhengjie Du, Zhu Ding, Linke Li, Qiushi Wu, Mathias Payer, and Bing Mao. Silent bugs matter: A study of compiler-introduced security bugs. In USENIX Security, 2023. [31] Meng Xu, Chenxiong Qian, Kangjie Lu, Michael Backes, and Taesoo Kim. Precise and scalable detection of double-fetch bugs in os kernels. In IEEE S&P, 2018. USENIX Association 33rd USENIX Security Symposium 1223 A Double Fetch Statistics In this section, we present detailed statistics related to fetch and double fetch occurrences across our benchmarks. Table 5presents our results. For both generic fetches and double fetches (i.e., fetches that transfer data overlapping with a previous fetch within the same syscall) we detail: the rate of (double) fetches relative to the total number of syscall samples collected for the benchmark (the Rate column), the total number of distinct syscalls that executed a fetch during the benchmark (the Syscalls column). Additionally, we report the minimum, average, and maximum number of (double) fetches executed by one single syscall sample. The last row in the table refers to syscalls executed by background applications. As shown in the table, each benchmark has a nontrivial (e.g., up to 21 for Phoronix) number of syscall types with double fetches. Moreover, the number of double fetches performed by double-fetch syscalls greatly varies (e.g., ranging from 1 to 218 for Phoronix). Table 5: Statistics for (double) fetch rates. Benchmark Statistic Fetches Double Fetches LMBench Rate 1/2 1/273457 Syscalls 38 7 Min/Avg/Max 1/1/459 1/50/67 OSBench Rate 1/3 1/80 Syscalls 17 5 Min/Avg/Max 1/6/134 1/42/43 Phoronix Rate 1/4 1/5993 Syscalls 47 21 Min/Avg/Max 1/1/661 1/1/218 Background Rate 1/2 1/644 Syscalls 93 20 Min/Avg/Max 1/2/4099 1/130/467 Across all benchmarks, syscalls are likely to execute fetches (around 1 in 3 syscalls fetch user buffers), but more often they will fetch only a small number of user buffers (e.g., on average syscalls fetch 6 user buffers on OSBench). This prompted us to use a linked list as the default data structure for efficient caching. However, across all benchmarks, there are occurrences of fetchheavy syscall executions (e.g., with up to 661 fetches on Phoronix), motivating the need for an adaptive strategy that resorts to a red-black tree to handle fetch-heavy scenarios. Moreover, the striking difference between fetch rates and double fetch rates across all benchmarks, with double fetches being far less frequent, suggests that cache insertions happen often and thus it is crucial to factor in insertion time when determining the optimal threshold to convert to a red-black tree. Looking more closely at the variability of the number of fetches a syscall makes—across the syscalls that fetch data in our profile—we observed that ≈ 55% have a stable fetch rate (i.e., they fetch the same number of user buffers every time). Most syscalls with stable fetch rates perform either 1, 2 or 3 fetches, with the exception of sendmmsg which performs 6 fetches all the time. From the syscalls that have variability in the number of executed fetches, 45 system calls execute between 1 and at most 8 fetches while only 5 system calls execute more than 8 fetches. Special cases are 3 system calls, i.e., pwrite64, execve, and write that have high variability and execute 1-255, 1-1411 and 1-4,099 fetches, respectively. Again, this variability motivates the need for an adaptive strategy for cache lookups. Looking instead more closely at the variability of the number of double fetches—across the syscalls that fetch data twice in our profile—we observed that ≈ 60% execute only one double fetch, while 33% execute either 1 or 2 double fetches and the sendmsg syscall may execute between 3 and 6 double fetches. The execve syscall is again an exception and can execute between 1 and as many as 467 double fetches. 1224 33rd USENIX Security Symposium USENIX Association