HSP-V: hypervisor-less static partitioning for RISC-V COTS platforms

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000. Digital Object Identifier 10.1109/ACCESS.2024.DOI HSP-V: Hypervisor-less Static Partitioning for RISC-V COTS Platforms JOÃO SOUSA1, JOSÉ MARTINS1, TIAGO GOMES1, AND SANDRO PINTO1 1Centro ALGORITMI / LASI - Universidade do Minho, Portugal (e-mail: [email protected]; [email protected]; mr[email protected]; [email protected]) Corresponding author: João Sousa (e-mail: [email protected]). This work has been supported by FCT - Fundação para a Ciência e Tecnologia within the R&D Units Project Scope UIDB/00319/2020, SFRH/BD/00297/2023, and SFRH/BD/138660/2018; and partially supported by the European Union’s Horizon Europe research and innovation program under the project Cross-platform Open Security Stack for Connected Devices (CROSSCON) with grant agreement No 101070537. ABSTRACT Virtualization technologies have played a pivotal role in consolidating Mixed-Criticality Systems (MCS) onto a single computing platform. However, not all RISC-V processors present in Commercial Off-The-Shelf (COTS) platforms feature the hypervisor extension, which poses a significant challenge in offering virtualization support. This paper introduces HSP-V, a ready-to-run low-level software stack to provide static partitioning on RISC-V COTS platforms lacking virtualization extensions. HSP-V leverages the Domain feature of the RISC-V Open Source Supervisor Binary Interface (OpenSBI) reference implementation to establish partitions using the capabilities provided by the Physical Memory Protection (PMP) unit. Additionally, it provides other capabilities such as interrupt partitioning, direct interrupt injection, cache partitioning, and platform-level isolation for DMA-capable devices. The conducted evaluation assesses the influence of HSP-V on different performance metrics, including domain boot time, interrupt latency, code size, and execution performance using the MiBench embedded benchmark. HSP-V achieves highly deterministic interrupt latency with an average execution time of 457 ns (with a standard deviation of only 22 ns), with essentially zero traps in the Domain execution. In scenarios with cache interference, the HSP-V keeps the performance overhead as low as 0.39% for the best case scenario. INDEX TERMS Mixed-Criticality Systems, Virtualization, Static-partitioning, RISC-V, OpenSBI. I. INTRODUCTION Cyber-physical systems have evolved significantly in the past few decades [1], transitioning from single-purpose devices with limited communications and simple interfaces to powerand compute-hungry general-purpose systems with multiple functionalities and complex interactions [2]. To meet the demands for reduced size, weight, power, and cost (SWaP-C), there has been a paradigm shift towards the deployment of mixed-criticality systems (MCS), which integrate and consolidate different applications with distinct levels of criticality into a single hardware platform [3]–[5]. This approach requires spatial, temporal, and fault isolation among all subsystems, ensuring that subsystems with lowercriticality do not compromise the timing, functionality, or performance of the safety-critical ones. Virtualization stands out as the key enabler technology for consolidating MCSs, focusing on workload consolidation and isolation between different computing environments, e.g., operating systems (OSes), on a single hardware platform. Hypervisors have been extensively used for virtualization, providing the ability to efficiently share resources, support different workloads according to the criticality level of the applications, and assure strong isolation between all instances. Currently, hypervisors can span from minimalist approaches optimized for safety and security, e.g., static partitioning hypervisors (SPHs) [6], to more feature-rich and resource-efficient solutions, such as Xen [7] and KVM [8]. SPHs allocate fixed and dedicated resources to each virtual machine (VM) at the time of configuration, including central processing unit (CPU) cores, memory, devices, and interrupts [6], [9]–[11]. This minimal approach, specially tailored for MCS, ensures the effective isolation and allocation of resources, which is mainly possible by leveraging instructionset architecture (ISA) virtualization extensions [10], [12]. Such extensions can already be found in well-established computer architectures such as Arm (since Armv7 [13]), Intel (introduced with Intel VT [14]), and more recently in RISC-V with the hypervisor extension specification [12]. VOLUME 1, 2024 1 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ The RISC-V ISA [15] has been gaining traction across a wide range of computing domains, including MCS [12], [16], [17]. In comparison to other architectures, RISC-V is highly modular, reserving part of the encoding to custom extensions, enabling highly specialized implementations that can scale from simple microcontrollers to supercomputers. Regarding virtualization support, the hypervisor extension [18], ratified in Q4 2021, is defined by the privileged architecture specification of RISC-V. Despite the extension being already supported in QEMU and several open-source soft-core RISC-V processors deployed in field-programmable gate array (FPGA) [9], [10], [19], its support in commercial off-the-shelf (COTS) platforms remains scarce, where only one silicon-based implementation is known to be available [20], limiting the widespread utilization of hypervisors with this architecture. Targeting RISC-V hardware platforms lacking virtualization support, this paper presents HSP-V, a hypervisorless static partitioning solution that allows the deployment of MCS systems in scenarios where using a hypervisor is not feasible. HSP-V is based on OpenSBI1, the de facto Supervisor Binary Interface (SBI) firmware implementation for RISC-V. By leveraging the OpenSBI Domain feature, a system-level partition of underlying hardware having dedicated memory regions and harts (i.e., hardware threads, essentially cores in RISC-V lingo), HSP-V can run at the highest privilege mode, i.e., machine mode, while keeping each partition running at the supervisor/user modes. Partitioning is achieved using memory isolation primitives widely available on RISC-V processors, such as the Physical Memory Protection (PMP) unit. Although OpenSBI domains already embody the core requirements of a static partitioning system, some features are still missing, hampering the fully deployment of the functionalities provided by an SPH: (i) interrupt partitioning and domain assignment; (ii) interVM interference mitigation; and (iii) platform-level memory isolation for direct memory access (DMA) devices2. The main contributions of this article are summarized as follows: •The introduction of a hypervisor-less static partitioning solution based on the OpenSBI reference implementation, specially designed for RISC-V COTS hardware platforms that lack the hypervisor extension; •Several contributions to the OpenSBI reference implementation, such as: (i) interrupt partitioning by mediating the access to the platform-level interrupt controller (PLIC); (ii) shared cache partitioning; and (iii) assignment of DMAcapable devices to different domains by using the platformspecific input-output memory protection Unit (IOMPU); •A comprehensive evaluation of the HSP-V regarding code size, boot time and performance overhead, interference, and interrupt latency. 1OpenSBI: https://github.com/riscv-software-src/opensbi 2OpenSBI support for technologies such as the IOPMP [21] is not yet available, but it is on the project’s roadmap. II. BACKGROUND A. PARTITIONING TECHNOLOGIES Partitioning technologies, such as TEEs and SPHs, play a pivotal role in modern computing systems. While a TEE provides a secure and isolated environment for sensitive operations and data with high levels of confidentiality and integrity, virtualization can be leveraged for workload consolidation and isolation between different computing environments on a single hardware platform. 1) Trusted Execution Environments (TEEs) A TEE involves splitting the system into two distinct worlds [22], i.e., a non-secure world mainly used for richOS support and applications, and a secure world that is commonly responsible for executing critical functionalities such as data encryption, fingerprint authentication, monetary transaction services, etc. Such secure services usually execute under a trusted application (TA) supported by a trusted OS. The main goal is to ensure that applications running in the secure world are protected and isolated from any interaction by any other component present in the system. Examples of TEE implementations include Intel SGX [23] and Arm TrustZone [24]. Intel SGX provides hardware-assisted trusted execution, creating secure enclaves to protect application code and data from any access from other software component, even those with root privileges. Similarly, Arm TrustZone offers hardware-based access control by enabling a processor to run in two isolated execution environments, i.e., the secure and non-secure world. TrustZone is widely used in mobile devices and ARM-based servers [25], hosting secure kernels such as Trustonic3, Qualcomm’s QSEE4, and Linaro’s OP-TEE 5, ensuring the protection of securitycritical data, and facilitating the deployment of various TAs with distinct functionalities. 2) Static Partitioning Hypervisors (SPHs) Modern SPHs include Jailhouse [26], Xen Dom0-less [27], and Bao [10]. They all follow a minimalist implementation, on the order of a few thousand Source Lines of Code (SLoC), and they mainly perform the partitioning and assignment (with no sharing) of platform hardware resources, such as CPU, memory, devices, and interrupts, among the existing VMs. Since each virtual CPU (vCPU) is pinned to a single physical CPU, SPHs do not include a scheduler as part of the hypervisor internals, achieving reduced size and complexity. All these features are mainly possible by leveraging dedicated ISA virtualization extensions [10], [12]. For instance, Arm’s hardware virtualization provides a new higher privilege mode for the hypervisor, offering support for interrupt virtualization, input–output memory management unit (IOMMU) that securely allows VMs to directly control DMA-capable devices [21], [28], [29], and the two-stage 3Trustonic:https://www.trustonic.com/ 4Qualcomm: https://www.qualcomm.com/products/snap-dragon/security 5OP-TEE: https://github.com/OP-TEE/ 2VOLUME 1, 2024 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ memory address translation. Nonetheless, this latter may represent a potential challenge to the security and real-time requirements of MCS. Besides logical space and temporal isolation, MCSoriented hypervisors must also take into account the shared micro-architectural resources present in complex memory hierarchies of modern multi-core platforms, e.g., last-level caches, interconnects, and memory controllers, as critical subsystems can be sensitive to the timing variations resulting from contention on such components [30]–[32]. To mitigate inter-core interference at the hypervisor level [33], several techniques have been proposed by the real-time research community, such as cache coloring [34]–[36], DRAM bank coloring [37], memory throttling [35], [38], [39], and I/O regulation [40], [41]. B. RISC-V RISC-V is an open-standard ISA created as a research project in 2010 at the University of California, Berkeley [15], and currently managed by the non-profit RISC-V International. With a highly permissive license that allows for both open and proprietary implementations, its highly flexible and modular design enables different features, e.g., floating point, atomic and vector instructions, etc., to be added as extensions on top of the base integer instruction set (both on 32-bit and 64-bit instructions). Within the scope of this article, there are some key components that, working together, are essential to provide a flexible and secure computing environment that allows the deployment of the HSP-V architecture. 1) RISC-V Privileged Modes The RISC-V privileged architecture specification [18] defines three base privilege modes (from higher to lower privilege): Machine mode (M-mode), Supervisor mode (S-mode), and User mode (U-mode). The only mandatory privileged level is the M-mode, commonly intended for hosting the firmware and that operates only with physical addresses, i.e., without virtual address translation. The S-mode and the U-mode are used to run OSes and applications, respectively. Typically, microcontrollers implement only the U-mode, while application class processors also implement the S-mode, which provides support for virtual memory and enables the execution of Unix-like OSes. In addition to these privileged levels, the privileged spec defines the hypervisor extension, which introduces the concept of supervisor virtualization mode by adding two orthogonal, but less privileged, modes: the Virtual-Supervisor (VS) and the Virtual-User (VU). Furthermore, the S-mode is extended with hypervisor functionalities such as control over two-stage translation and renamed Hypervisor-extended Supervisor mode (HS-mode) [12], [42]. While QEMU and several open-source soft-core RISC-V processors deployed in FPGA already support the hypervisor extension, its adoption in COTS platforms is currently limited [20]. Table 1 summarizes the landscape of widely used linux-capable RISC-V COTS platforms that lack the hypervisor extension. Nonetheless, several security features TABLE 1: RISC-V Linux-capable platforms without the hypervisor extension. Platform SoC Security Features Interrupt Controller PolarFire SoC Icicle Kit PolarFire SoC FPGA PMP, MMU, IOMPU, Waymasking PLIC BeagleV-Ahead Alibaba T-Head TH1520 SoC PMP, MMU, OTP, TEE System PLIC SiFive HiFive Unleashed SiFive Freedom U540 SoC PMP, MMU, OTP, Waymasking PLIC SiFive Unmatched SiFive Freedom U740 SoC PMP, MMU, OTP, Waymasking PLIC Nezha Allwinner D1 SoC PMP, MMU, IOMMU PLIC VisionFive StartFive JH7100 64-bit Soc PMP, MMU, OTP, TRNG PLIC VisionFive 2 StartFive JH7110 64-bit Soc PMP, MMU PLIC Lichee RV Dock Allwinner D1 SoC PMP, MMU PLIC are still supported, e.g., PMP, MMU, IOMPU (available in the PolarFire SoC Icicle Kit for protecting DMA-capable devices), WayMasking (available in the PolarFire SoC Icicle Kit, in the Sifive Hifive Unleashed and in the Sifive Unmatched for cache partitioning) and others. For the interrupt controller, they all implement PLIC, which provides no interrupt partitioning or virtualization support. 2) Physical Memory Protection (PMP) The RISC-V provides a memory protection mechanism called PMP [18] that is capable of limiting supervisor and user (and optionally machine) mode accesses to the physical memory address space. For this reason, when virtual memory is present (enabled by the MMU), and a translation is needed, the PMP takes only effect after the memory translation. PMP is controlled from M-mode, allowing the definition of a whitelist for address space regions, each with different access rights (i.e., read, write, and execute) by configuring a set of Control and Status Registers (CSRs). Depending on the implementation, the PMP unit can use either 16 or 64 CSRs, thus limiting the maximum number of currently accessible memory regions. An access to a memory address not included in the whitelisted memory regions, or that violates its permissions, will cause an access fault and subsequent trap to M-mode. 3) Interrupts and PLIC The RISC-V architecture includes three main classes of interrupts: (i) software interrupts, comparable to inter-processor interrupt (IPI); (ii) timer-based interrupts; and (iii) external interrupts. While softwareand timer-based interrupts are considered local interrupts and managed by per-hart interrupt controllers, such as the Core-local Interrupt (CLINT6) or the Core-local Interrupt Controller (CLIC7), external interrupts are platform-wide and shared among all harts. The PlatformLevel Interrupt Controller (PLIC8) is responsible for routing and multiplexing peripheral interrupts to all harts in the system, depending on how it is configured through an MMIO interface. The PLIC is able to multiplex up to 1023 distinct external interrupts to one or more hart contexts, i.e., a combination of a hart and its associated privilege level. As only Mand S-mode can receive interrupts, the PLIC typically has two contexts per-hart. 6CLINT: https://github.com/pulp-platform/clint 7CLIC: https://github.com/riscv/riscv-fast-interrupt 8PLIC: https://github.com/riscv/riscv-plic-spec VOLUME 1, 2024 3 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ There are two main MMIO regions in the PLIC: (i) a global configuration region; and (ii) a per-context region for defining the harts priority mask and handling interrupts. Although the PLIC has been the standard interrupt controller for RISC-V since its inception, the new Advanced Interrupt Architecture (AIA)9[43] is now the reference interrupt controller that will supersede the PLIC. The AIA controller includes a redesigned PLIC, called the Advanced PLIC (APLIC), which despite including the very same functionalities, it does not provide backward compatibility, i.e., systems or applications designed to work with the original PLIC are not compatible with the APLIC. Despite the emergence of AIA, current Linux-capable RISC-V COTS platforms still rely on the PLIC, as seen in Table 1. C. OPENSBI The RISC-V non-ISA SBI specification aims at providing an abstraction over low-level, implementation-defined, and platform-level components and mechanisms to ease the implementation and porting of supervisory software. It defines a number of run-time services meant to be provided by M-mode firmware, such as hart-state management, IPIissuing, TLB invalidation, and shootdown. The OpenSBI project is an open-source reference implementation of the RISC-V SBI designed to be highly modular and easily adaptable to a wide range of RISC-V platforms, supporting different ISA extensions and non-ISA components. It can be directly used as the run-time firmware (or as a library included in external firmware or bootloaders), supporting the handling of misaligned memory accesses and the emulation at M-mode of Extensions that are not implemented (e.g., Legacy and IPI Extension [44]), required by supervisor or user software. The OpenSBI Domain system, is capable of partitioning the underlying hardware by assigning dedicated memory regions to one or more harts. Besides partitioning harts and memory/MMIO regions, OpenSBI also restricts the effect of the services it provides to the invoking domain’s harts. For example, it will deny requests to send IPIs to harts which are part of other calling hart’s domain. Figure 1 depicts an example OpenSBI domain system configuration comprising two domain instances running in S-mode: (i) one with a Unix-like OS with its applications running in U-mode; and (ii) the other with an RTOS configuration (FreeRTOS). The initial boot stages are responsible for loading both OpenSBI’s and the domains’ images to the main memory, being the configuration passed in the form of a Device Tree (DT) node following a custom binding. If this node is not present, OpenSBI assumes a single "root" domain containing all harts, devices, and memory (excluding its own memory). Next, the OpenSBI domain instances are created with their harts and memory regions with respective access permissions, followed by some sanity checks to avoid any user misconfiguration such as domains’ memory overlapping. Finally, 9AIA:https://github.com/riscv/riscv-aia Hart 1 M S PMP APPAPPAPP OpenSBI U PLIC Dev nDev n-1Device 1 H n-1... H n ... GPOS RTOS DOMAIN 2DOMAIN 1 FIGURE 1: GPOS and RTOS configuration with vanilla OpenSBI. it assigns each memory to its domains through the PMP entry setup and jumps to a pre-configured address in the domain’s boot hart. The other domain’s harts may be later woken up via the hart power-state management service. D. CHALLENGES OF STATIC PARTITIONING WITHOUT VIRTUALIZATION EXTENSIONS ON RISC-V COTS. Designing a static partitioning solution without relying on virtualization extensions is not directly possible on current available RISC-V COTS platforms. Despite OpenSBI already implementing the core functionalities towards the goal of static partitioning with the domains system, the most important challenges still need to be addressed: •Interrupt partitioning by mediating a domain’s access to the PLIC: Some PLIC registers include the configuration of multiple interrupts and contexts, with a few registers being shared between both domains. The shared PLIC address space across domains, as depicted in Figure 1, highlights the challenge of preventing one domain from interfering with the interrupts of adjacent domains. Additionally, it is critical to provide mechanisms to allow the execution of OSes with unmodified PLIC drivers (e.g., with trap-and-emulate). •Assignment of DMA-capable devices to different domains via the IOMPU: It is mandatory to provide memory isolation at the system-level, i.e., including DMA devices in isolated domains. •Shared cache partitioning: It is necessary to deploy mechanisms to mitigate inter-hart interference, namely contention for shared cache lines. The following sections detail how these contributions were added to the OpenSBI using the Microchip’s PolarFire SoC FPGA Icicle Kit [45], a widely-available RISC-V hardware platform featuring the five-core Linux capable RISC-V microprocessor subsystem. 4VOLUME 1, 2024 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ DOMAIN 1 DT Device ID=10 Complete Threshold Mem-regions DOMAIN 2 Device ID=35 Complete Threshold Mem-regions Domain 2 visibility Domain 1 visibility Application1 Application2 Pending Enable Claim/Complete Threshold Claim/Complete Threshold Priority Application1 Application2 Pending Enable Claim/Complete Threshold Claim/Complete Threshold Priority Application1 Application2 ID=35 ID=10 ID=35 ID=10 FIGURE 2: Two HSP-V domain systems and their respective memory access permissions. III. HSP-V IMPLEMENTATION A. OPENSBI DOMAIN CONFIGURATION ENHANCEMENTS Domain configuration is done by adding an opensbi-domains node under the chosen node to the platform’s hardware description DT file. Listing 1 includes the settings for the configuration illustrated in Figure 2, which is composed of two bare-metal application domains, each statically pinned to a single CPU and to a single device. The custom binding for this node includes two types of subnodes: memory regions and domain instances. A memory region node essentially defines a base address (base) and a size (order) which may refer to actual memory size or MMIO regions, while a domain instance defines a domain’s configuration with four important properties: (i) possible-harts, (ii) regions, (iii) possibledevices, and (iv) cache-partitions, which are detailed below. Other domain instance node properties include (i) the boothart, (ii) the next-mode (next privileged level), and (iii) the next-addr (entry point address) for the domain10. possible-harts: this property contains the pointer handle (phandle) for each hart assigned to the domain. In this specific configuration, Domain1 is assigned to cpu1 and has total access (read, write, and execute permissions) to its application memory region (Dom1MainMem), while Domain2 is assigned to cpu3 and has total access to its distinct application memory region (Dom2MainMem). regions: this property defines the physical address space carvings assigned to the domain with an array of tuples, each containing a phandle to the memory region node, plus a bitmap for the assigned permissions (read, write, execute). possible-devices: this property contains an array of phandles to devices assigned to that domain instance. This property 10Vanilla OpenSBI Domain configuration: https://github.com/riscvsoftware-src/opensbi/blob/master/docs/domain_support.md 1 chosen{ 2 opensbi−domains { 3 Dom1MainMem: Dom1MainMem { 4 compatible ="opensbi,domain,memregion"; 5 base =<0x0 0x8020000>; 6 order =<20>; 7 }; 8 Dom2MainMem: Dom2MainMem { 9 compatible ="opensbi,domain,memregion"; 10 base =<0x0 0x8010000>; 11 order =<20>; 12 }; 13 Dom1instance: Dom1instance { 14 compatible ="opensbi,domain,instance "; 15 possible −harts =<&cpu1>; 16 regions =<&Dom1MainMem 0x7>; 17 possible −devices =<&periph_with_id10>; 18 cache− partitions =<0x0F>; 19 ... 20 }; 21 Dom2instance: Dom2instance { 22 compatible ="opensbi,domain,instance "; 23 possible −harts =<&cpu3>; 24 regions =<&Dom2MainMem 0x7>; 25 possible −devices =<&periph_with_id35>; 26 cache− partitions =<0xF0>; 27 ... 28 };};}; Listing 1: HSP-V configuration for the system in Figure 2. was added to the vanilla OpenSBI configuration since a domain instance lacks device interrupt details and DMAcapable device’s information. In the default configuration, to assign a device to a domain instance, it was necessary to create a dedicated memory region for that device. However, this information can be directly retrieved from the device node reg property. Thus, in the new configuration, the domain instantiation is able to retrieve the memory region for the device and its associated interrupts (assigning them to the partition as discussed in Section III-B), and to identify DMA-capable devices and their respective IOMPU ID (as discussed in Section III-D). This optimized device MMIO region assignment mechanism is able to streamline and simplify the configuration of domains, reducing the possibility of user misconfigurations. cache-partitions: This property represents a bitmap for the cache partitions assigned to a given domain, independently of the method used to partition the cache. For this specific configuration, the cache partitioning is done through domains by assigning four cache ways to Domain1 (cache-partitions property with 0x0F), and different four cache ways for Domain2 ( cache-partitions property with 0xF0). B. INTERRUPT PARTITIONING When devices are assigned to domains, these must access the PLIC to configure and handle respective device’s interrupts. However, concurrent and unsynchronized accesses to PLIC registers might result in unpredictable behaviour for the involved domains. Even if different domains cooperate to perform such accesses, this would still be a major VOLUME 1, 2024 5 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ security/safety vulnerability, as a malicious domain could intentionally interfere and tamper with other domains’ interrupts. To prevent this, a PLIC partitioning mechanism in OpenSBI was implemented. The approach is based on the principle that, as explained in Section II-B, PLIC context MMIO regions are specific to a given hart, while the global configuration regions must be shared among all domains. The implemented mechanism starts by verifying if no PLIC MMIO regions are defined in the regions properties, followed by configuring the PMP to allow domain access to its harts supervisor context MMIO region. As for the global configuration register, and for a realistic number of device interrupts and due to the limited number of PMP CSRs, it would be impossible to configure the PMP to grant access to the registers which pertain only to those interrupts. Furthermore, some registers configure multiple interrupts simultaneously on a per-bit basis - this protection granularity cannot be enforced by the PMP - and, therefore, to protect and mediate access to this critical PLIC region (memory regions represented with color red in Figure 2, i.e., the Priority,Pending and Enable PLIC regions), the proposed mechanism uses the classical trap-and-emulate technique. Since the RISC-V access control faults generate precise exceptions, when a domain tries to access the global PLIC region it traps to M-mode. OpenSBI uses the exception information CSRs (e.g., mcause,mtval,mepc) to read the fault instruction and decode the access to retrieve key information such as the access type (load or store), the destination/source register, and the access width. Since the exception program counter (mepc) carries information about the virtual address, it is required to set the mstatus.MPRV bit to read the instruction. When this bit is enabled, M-mode memory accesses are executed as if they were coming from the privilege level that generated the trap. Hence, when the domain has virtual memory enabled, the access is subject to address translation using the domain’s page tables. For the accessed address available through mtval, if virtual memory is enabled, it is necessary to perform a manual pagetable walk to retrieve the actual physical address. Then, if this accessed physical address is indeed part of the critical PLIC region, the OpenSBI invokes the PLIC emulation routines. Based on that address, the type of PLIC register being accessed is decoded. This access is then passthrough (or ignored) based on the accessing domain contexts and assigned interrupts. At the end of this process, the execution returns to the previous execution context and resumes from its last instruction. Nonetheless, trap-and-emulating this PLIC region will only result in significant overheads when configuring interrupts. These MMIO registers are not on the critical path for the interrupt handling, as they are typically only accessed during domain’s initialization. The interrupts are still delivered directly to S-mode, and the PLIC registers touched during interrupt handling are directly accessible to domains without any traps. Therefore, the proposed approach to interrupt partitioning does not cause any noticeable overheads in the interrupt latency. C. CACHE PARTITIONING In the realm of MCS, to comply with security and real-time requirements, minimizing deviations in the execution time is crucial for maintaining a deterministic behavior. However, contention at inter-hart (therefore at inter-domain) memory hierarchy could result in significant and unpredictable execution time, i.e., at shared micro-architectural resources. Specifically, regarding shared Last-Level Caches (LLCs), a given domain executing a memory intensive workload might inadvertently evict the cache lines of a critical domain, incurring in high memory access latency and low memory bandwidth, and potentially resulting in missing the execution deadlines. In a worse case, a malicious domain might intentionally evict such lines to perform Denial-of-Service attacks (DoS) [46], [47] or even apply cache-side timing channel techniques (e.g., Prime+Probe [48], [49]) to retrieve information on the victim domain’s data or execution flow. The Microchip’s Polarfire SoC features a PhysicallyIndex/Physically-Tagged (PIPT) 2MiB shared and a unified L2 LLC following a 16-way set-associative topology [45]. Besides allowing the use of carve-outs directly as scratchpad memories, with essentially constant access times, it also provides a mechanism to lock cache ways with a per-master granularity, where each hart has two masters one for the instruction cache, and another for the data cache. A cache controller interface provides a WayMask register for each master, where each bit in the mask corresponds to one of the cache ways [45]. When a bit is clear in a master’s mask, it indicates that this specific way cannot be evicted by that master. Nonetheless, this mechanism does not provide any logical isolation, as a hart can still read its masked ways. For the isolation requirements, the PMP unit is always needed. Thus, this locking mechanism is used to partition the LLC among the multiple domains according to the cachepartitions property of the domain’s DT binding, i.e., a bitmap representation of the assigned cache partitions to that domain. Given the way-locking mechanism available in the Microchip’s Polarfire SoC, each of the 16 least-significant bits in the cache-partitions property represents eviction rights over one of the cache ways. At initialization time, and for each domain, the WayMask registers are set for the assigned harts instruction and data caches with the value of cache-partitions. The exception is the case when cache-partitions is not set in the domain’s configuration. Hence, it is assumed that all cache ways are assigned to all domain’s harts and consequently shared among them. Despite the advantages of including the WayMask register in the cache controller interface, regarding the DMA devices it essentially groups the different DMA channels into a single WayMask master. As a result, it is not possible to achieve a fully partitioned cache for certain device assignment combinations, i.e., locking ways for a specific DMA channel, will also lock the same ways for other channels allocated to adjacent domains. 6VOLUME 1, 2024 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ TABLE 2: SLoC and binary size (bytes). SLoC Size (bytes) c asm Total .text .data .bss Total OpenSBI utils 6236 27 6263 41597 1240 13224 56061 platform 373 0 373 2247 192 544 2983 sbi 8507 229 8736 52010 816 128376 181202 firmware 33 860 893 49695 120 0 49815 Total 15149 1116 16265 145549 2368 142144 290061 HSP-V utils 7159 27 7186 44532 1240 15624 61396 platform 480 0 480 2273 320 40 2633 sbi 9130 229 9359 54680 824 128376 183880 firmware 33 863 896 49691 120 0 49811 Total 16802 (+10.9%) 1119 (+0%) 17921 (+10.2%) 151176 (+3.8%) 2504 (+6%) 144040 (+1.3%) 297720 (+2%) D. DMA PROTECTION Unmediated access to the main memory by a domain’s nonCPU bus master (i.e., a DMA-capable device) can result in data corruption (and thus the state and/or sensitive information) of other domains. To avoid this, the Microchip’s Polarfire SoC includes a built-in IOMPU for each of these masters, including Ethernet, eMMC, and USB peripherals [45]. The IOMPU configuration registers essentially follow the same structure as the PMP CSRs. However, the number of regions for each IOMPU device varies from 2 to 16, e.g., 4 configuration entries for the MMC master block and 8 entries for Ethernet master blocks. As a result, since a domain is a set of harts and memories and each hart supports a maximum of 16 regions (i.e., 16 PMP entries), the number of regions for a given block master might be less than the number of regions assigned to its domain. Facing this, the HSP-V approach configures the memory regions of these masters accordingly to the domains that will leverage the device. If there are still not enough registers to configure the domain’s region, a fault is triggered and the system is fully halted before starting any domain. At runtime, in case a peripheral tries to access a region not present in its IOMPU regions, an interrupt is issued to OpenSBI, which acts by halting all harts belonging to a device’s domain. IV. EVALUATION The evaluation of the HSP-V was conducted on a Microchip PolarFire SoC Icicle Kit board [45], which features a SiFive E51 platform management hart, a quad-core U54 application cluster with per-core 32 KiB L1 data and instruction caches, and a 2 MiB shared L2 cache. The performed tests include HSP-V code size, boot overhead, execution performance and inter-domain interference, and interrupt latency. A. CODE SIZE This work extends the OpenSBI v1.0, adding significant features provided by HSP-V while maintaining the original code structure. Table 2 presents the SLoC and the final binary size of subsystem for both the vanilla OpenSBI and HSP-V, retrieved with the compiler optimizations set to -O2. The HSP-V adds about 1656 SLoC to the 16265 SLoC of the vanilla OpenSBI, which corresponds to an increase of around 10%. Most of the additional SLoC are in (i) the utils directory, specifically, in the DT parsing logic; (ii) the platformdependent code with the driver implementation for applying the cache partitioning and the IOMPU register setup; and OpenSBI init Domain (ii) Domain (i) S-mode M-mode OpenSBI HSP-V FreeRTOS boot T&E FreeRTOS boot Linux boot T&E Linux boot FreeRTOS boot time Linux boot time OpenSBI init FIGURE 3: Boot sequence for a configuration with two domains under the same platform. TABLE 3: Boot time (ms) of vanilla OpenSBI and HSP-V. Scenario OpenSBI init. time (ms) Total boot time (ms) avg std-dev avg std-dev OpenSBI freertos 80.134 0.303 94.140 0.748 linux 6734.653 5.026 freertos lock 80.202 0.200 94.049 0.200 linux lock 6800.564 6.043 HSP-V freertos 112.051 (+40%) 0.197 154.517 (+63%) 0.438 linux 8375.438 (+24%) 5.510 freertos lock 112.102 0.205 154.385 0.207 linux lock 8417.998 5.358 (iii) the sbi core that was enhanced with the PLIC trapand-emulation code. On the other hand, the final binary file with the features added by the HSP-V is around 298 KiB, which corresponds to an additional 8 KiB (mostly on the .text section) to the original OpenSBI binary file (290 KiB). Despite not completely negligible, the modifications required by the HSP-V dot not significantly impact the system’s Trusted Code Base (TCB). B. BOOT OVERHEAD This evaluation consists in measuring the total boot time of a configuration with two single-hart domains that follows the boot sequence illustrated by Figure 3. Domain (i) consists of a FreeRTOS configuration with a binary size of 57 KiB, while Domain (ii) includes a Linux-based system with an image size 104 MiB. The measurements include the total boot time (label: OpenSBI init) of both the HSP-V with the enhanced version of the OpenSBI, and the vanilla OpenSBI, as well as the boot execution time (labels FreeRTOS boot time and Linux boot time) of each domain, both represented by red arrows. Additionally, it was measured the execution time of the trap-and-emulation (label T&E) mechanism, which corresponds to the PLIC accesses for interrupt partitioning after setting up the domains, as well as the influence of enabling the cache partitioning feature. To carry out these measures (in clock cycles) the rdcycle pseudo-instruction was used, and the collected results are summarized in the Table 3. Regarding the initialization time without the cache lock, the vanilla OpenSBI takes on average 80.134 ms to complete, while the OpenSBI with the HSP-V requires 112.051 ms to finish the initialization, corresponding to a boot time overhead of around 40%. With the cache locking mechanism enabled, these values further increase to 80.202 ms for the vanilla OpenSBI, and 112.102 ms for the HSP-V. For the total VOLUME 1, 2024 7 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ FIGURE 4: MiBench automotive benchmark suite performance results. boot time without cache locking, the HSP-V requires around 154.517 ms for booting the FreeRTOS, and 8375.438 ms for booting the Linux system. These values, when compared to the native versions of both domains, correspond to an overhead of nearly 63% and 24%, respectively. This is mainly due to the trap-and-emulating operations required by the HSP-V for configuring the PLIC MMIO regions, which are not required in the vanilla OpenSBI. C. PERFORMANCE OVERHEAD AND INTERFERENCE To assess the performance overhead and the inter-hart/interdomain interference, it was used the MiBench Embedded Benchmark Suite’s automotive subset, a reference benchmark widely used in the evaluation of MCS [6], [10], [12], and the LMbench [50], a suite of portable micro-benchmarks designed to measure various aspects of a computing system’s performance. MiBench. This benchmark suite consist of six different tests that execute in a single-hart Linux-based domain, containing four memory-intensive algorithms susceptible to interference caused by the LLC and memory contention, such as qsort, susan corners, and susan edges. The interference between harts/domains is introduced by a bare-metal application that runs on other three harts and executes a memory-intensive workload that continuously performs sequential writes to a 1.5 MiB array with a stride equal to the cache line size (64 bytes). Each benchmark executed for four different system configurations: (i) hosted execution (solo), (ii) solo with cache locking enable (solo-lock), (iii) hosted execution under interference from multiple domains (interf), and (iv) interf with cache locking enable (interf-lock). For the tests including the cache locking mechanism, four cache ways (512 KiB) were allocated to the bare-metal application, and eight cache ways (1 MiB) to the Linux-based domain. The last four remaining cache ways (512 KiB) are reserved to be used as scratchpad memory by OpenSBI. Figure 4 depicts the performance results using the solo configuration as the baseline, where each bar represents the average execution time of 1000 samples. By enabling the cache partitioning (solo-lock), the overall performance decreases when compared with the solo configuration, which can be explained by the decreasing of the amount of available cache memory that is allocated to each domain. When stressing the system with interference (interf) caused by the bare-metal application running on the three remaining harts, the performance starts decreasing, especially in the memory-intensive benchmarks, i.e., the qsort small takes around 95.50 ms to complete (+50%), the susan corners small requires around 20.82 ms (+79.73%), and the susan edges small takes nearly 22.80 ms (+71.27%) to finish. With the interf-lock configuration, the cache partitioning mechanism mitigates the effect of this interference, which reduces the execution time of the previously mentioned memoryintensive benchmarks, i.e., the qsort small takes now around 78.90 ms to complete (+25.72%), the susan corners small requires now around 15.16 ms (+30.87%), and the susan edges small takes nearly 16.89 ms (+26.86%) to finish. Overall, the benchmarks handling smaller data sets (-small) are more susceptible to cache interference than the large versions. LMBench. This benchmark suite targets UNIX systems and aims at measuring various aspects of a computer system’s performance, such as memory latency and bandwidth, context switching, file system operations, and inter-process communication, among others. This evaluation only uses the bw_mem benchmark, which was used to evaluate memory operations bandwidth for different block sizes, i.e., 512KiB, 1MiB, and 1.5MiB, executed for the same system configurations as MiBench, i.e., for solo,solo-lock,interf, and interflock. The interference was caused by the same bare-metal (for interf configurations), and the cache locking mechanism followed the same way allocation as for MiBench, i.e., four cache ways to the bare-metal application and eight cache ways to the Linux-based domain. Figure 5 depicts the performance results using the solo configuration as the baseline, where each bar represents an average memory bandwidth in megabytes per second (MiB/s) of 100 samples. For each sample, the micro-benchmark was configured with 10 warmups and 1000 repetitions (–W 10 –N 1000), encompassing 100000 samples (per bar). LMBench results reinforce the same conclusions as MiBench, with the behaviour of solo,solo-lock,interf and interf-lock configurations following the same pattern. Nevertheless, the results show that the relative performance of the system decreases with the increase of the workload 8VOLUME 1, 2024 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ FIGURE 5: LMbench automotive benchmark suite performance results. (except for the copy operations). For the copy operations, the relative performance degradation can be explained by the workload being equal (512KiB) or higher (1 and 1.5 MiB) than the available LLC cache for each domain. As the copy operation uses two buffers (the source and destination buffers are cacheable), the size of necessary memory doubles (e.g., the workload of cp is two times the size of wr), making it the most memory-intensive micro-benchmark. For 512 KiB workload, the memory bandwidth rates are 606 MiB/s in cp, 390 MiB/s in fcp, and 471 MiB/s in bcopy; for 1 MiB workload the memory bandwidth rates are 233 MiB/s in cp, 191 MiB/s in fcp, and 208 MiB/s in bcopy; and for 1.5 MiB workload the memory bandwidth rates are 144 MiB/s in cp, 127 MiB/s in fcp, and 135 MiB/s in bcopy. Other experiments were performed with bigger memory workloads (from 256KiB to 2MiB). However, the achieved results followed the same pattern. D. INTERRUPT LATENCY To measure the interrupt latency, a crafted minimal baremetal benchmark application leverages an external timer peripheral configured in decrement mode with a 10 ms autoreload period, to both trigger the interrupts and measure their respective delay. All measurements were taken with cold L1 caches, which, between each measurement, are invalidated with the ifence instruction and the data flushed by reading the content of a dummy array with the size of the cache. Figure 6 depicts the results in the form of 5000 samples histogram for two configurations: (i) the interrupt latency of the vanilla OpenSBI without PLIC partitioning (Figure 6a); and (ii) the interrupt latency in the HSP-V with PLIC partitioning (a) HSP-V interrupt latency (with PLIC partitioning). (b) OpenSBI interrupt latency (without PLIC partitioning). FIGURE 6: Interrupt Latency overhead. (Figure 6b). The obtained results show that the HSP-V do not impact the interrupt latency, displaying a standard deviation of only 22 ns, with an average execution time of 457 ns. Such results correlate with what was previously explained in Section III-B, showing that the PLIC partitioning only causes traps to OpenSBI on interrupt configuration and not on interrupt handling, as external interrupts continue to be directly delegated to the S-mode in the mideleg CSR. V. HSP-V IN PERSPECTIVE WITH RELATED WORK This section provides an overview of existing RISC-V static partitioning systems, such as Bao [10], Jailhouse [26], XtratuM [51], Dom0-less (Xen) [27], Multizone [30], Keystone [52], and VOSySmonitoRV [16], putting them in perspective with the HSP-V solution. Table 5 highlights their differences considering the following features: (i) Virtualization Extensions support; (ii) the partition technology adopted, i.e., TEEs, Hypervisors, or other approaches exploring hardware RISC-V security primitives to statically isolate resources across several environments; (iii) the security design features; (iv) and the software license. Hypervisor technologies. The most prominent open-source SPHs supporting the RISC-V architecture (thanks to softcore implementations such as Rocket [12], CVA6 [42], and NOEL-V [53], deployed in FPGA) are Bao [10], Jailhouse [26], Xen dom0-less [27], and XtratuM [51]. Their static partitioning design defines CPU and IO memory accesses among all existing VMs. It adopts a 1-1 mapping of virtual to physical CPUs, with no need for a scheduler to mediate CPU allocation and ensure deterministic performance VOLUME 1, 2024 9 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3399601 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/