Zeyu Mi

Intrusion Detection Systems (IDSes) are widely employed to identify potential attacks in guest virtual machines (VMs). Nonetheless, traditional IDSes fall short of the demands of high-performance clouds. First, monitoring VM events increases the tail latency of guest services. Second, the throughput of traditional IDSes cannot meet high-performance cloud requirements, leading to event loss and reduced detection accuracy. Finally, cloud providers typically run complex IDS tools within the VM. Updating IDS functionality requires modifying guest VMs, which hurts maintainability. To overcome these challenges, this article presents EIDS, a cloud IDS framework with high performance and good maintainability. We observe that the main bottleneck is collecting VM status, and the collected status can be divided into fundamental and supplementary status. EIDS then splits the status collection procedure spatially and temporally. First, we provide a status monitor with a separate architecture that isolates the status collection logic in a microVM, thus minimizing the code in guest VMs and improving maintainability. Second, EIDS introduces a two-phase status collection method to handle multiple events in batches, asynchronously, for high IDS throughput. A tiny tracer, implemented with eBPF, operates inside the user VM to collect fundamental status. The complex status collector runs in an isolated microVM. It utilizes Virtual Machine Introspection (VMI) to gather supplementary status, using the fundamental status to bridge the semantic gap. The status collector batches the collection for multiple events to amortize the fixed overhead of microVM switching and improve event tracing throughput. Finally, to minimize tail latency overhead, a fine-grained and workload-aware scheduler executes IDS logic with small time slices during user VM idle periods. We implemented a prototype of EIDS in Linux-KVM and conducted a comprehensive evaluation. We compared EIDS’s performance with Falco, an open-source IDS widely used by Kubernetes and AWS for runtime security monitoring. The results demonstrate that, compared to Falco, EIDS reduces the 99 ^th -percentile latency overhead by 97% and achieves a 13.8X improvement in IDS event handling throughput.

As cloud tenants increasingly adopt ARM servers, securing their sensitive workloads becomes a critical concern. Confidential virtualization, which shields data from privileged software by establishing isolated VMs, has emerged as a standard approach to address this concern. However, ARM's official solution, the Confidential Compute Architecture (CCA), fails to meet this demand. It lacks support on most deployed hardware, and its widespread adoption by major cloud providers remains distant. In this paper, we propose TwinVisor, a novel system that enables confidential virtualization on ARM platforms. TwinVisor innovatively retrofits the mature ARM TrustZone to enforce strong isolation for confidential VMs, while minimizing the TCB by decoupling resource management from security mechanisms. Specifically, TwinVisor introduces a tiny hypervisor in the secure world (called S-visor in this paper) dedicated to protection, while reusing a full-fledged hypervisor (called N-visor) in the normal world for resource management. By combining S-EL2 hardware virtualization features with software-based same-privilege protection techniques, TwinVisor achieves broad compatibility, enabling its deployment on almost all ARM platforms. We have implemented two prototypes of TwinVisor, and our evaluation demonstrates that it can effectively protect confidential VMs while imposing negligible performance overhead for real-world applications.

Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language Models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8% and 88.2% throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining-an idea inspired by the CPU instruction pipelining-thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (<19.6% in throughput) across various LLM sizes, from 13B to 175B. PipeLLM's source code is available at https://github.com/SJTU-IPADS/PipeLLM.

Device pass-through has been widely adopted by cloud service providers to achieve near bare-metal I/O performance in virtual machines (VMs). However, this approach requires static pinning of VM memory, making on-demand paging unavailable. The hardware device I/O page fault (IOPF) capability offers an optimal solution to this limitation. Current IOPF approaches, using either standard IOMMU capabilities (ATS+PRI) or devices with independent IOMMU implementations, have not gained widespread adoption in public Infrastructure-as-a-Service clouds. This is due to high costs, platform dependency, and significant impacts on performance and service level objectives (SLOs). We present the Virtualized Page Request Interface (VPRI), a novel IOPF system developed through software-hardware collaboration. VPRI is not only platform-independent, free from address translation complexities, but also cost-effective, and designed to minimize SLO impact. Our work enables large-scale deployment of IOPF capability in Alibaba Cloud with negligible impact on SLOs. When integrated with memory management software, it significantly enhances memory utilization in public IaaS clouds, effectively overcoming the static memory pinning restriction associated with pass-through devices.

This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69× while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.

Nowadays, device passthrough I/O virtualization technology has played an essential role in cloud scenarios like network connection. However, the absence of widespread support for I/O Page Fault (IOPF) in I/O devices means that virtual machines (VMs) with directly assigned devices must pin all memory during startup, severely impacting memory efficiency and startup duration. Previous solutions addressing this issue suffered from various drawbacks, including reliance on specialized hardware, requiring VM modification, or imposing significant performance overhead. This paper introduces IOGuard, a software-based solution that universally, transparently, and efficiently removes the need for pinning all memory through the interception and analysis of I/O commands. Evaluation results demonstrate that IOGuard can support up to 16 VMs with a single host CPU core and incurs less than a 4% I/O performance overhead, substantially enhancing memory utilization for real-world workloads and significantly reducing VM startup times.

Today's cloud platforms offer large virtual machine (VM) instances with multiple virtual CPUs (vCPU) on manycore machines. These machines typically have a deep memory hierarchy to enhance communication between cores. Although previous researches have primarily focused on addressing the performance scalability issues caused by the double scheduling problem in virtualized environments, they mainly concentrated on solving the preemption problem of synchronization primitives and the traditional NUMA architecture. This paper specifically targets a new aspect of scalability issues caused by the absence of runtime hypervisor-internal states (RHS). We demonstrate two typical RHS problems, namely the invisible pCPU (physical CPU) load and dynamic cache group mapping. These RHS problems result in a collapse in VM performance and low CPU utilization because the guest VM lacks visibility into the latest runtime internal states maintained by the hypervisor, such as pCPU load and vCPU-pCPU mappings. Consequently, the guest VM makes inefficient scheduling decisions. To address the RHS issue, we argue that the solution lies in exposing the latest scheduling decisions made by both the guest and host schedulers to each other. Hence, we present a cooperative para-virtualized scheduling framework called CPS, which facilitates the proactive exchange of timely scheduling information between the hypervisor and guest VMs. To ensure effective scheduling decisions for VMs, a series of techniques are proposed based on the exchanged information. We have implemented CPS in Linux KVM and have designed corresponding solutions to tackle the two RHS problems. Evaluation results demonstrate that CPS significantly improves the performance of PARSEC by 81.1% and FxMark by 1.01x on average for the two identified problems.

The confidentiality of the operating system kernel addresses is crucial to keeping the kernel secure from malicious users. To avoid leaking this position, researchers have proposed various techniques to defeat the exploits of abnormal data flows embedded in the kernel’s memory safety loopholes, such as uninitialized memory read and buffer over-reads. However, this is far from complete. The kernel address can be leaked even in normal data flows without exploiting memory safety loopholes. We have designed a static analysis tool named Hawkeye to fill the gap. It searches for kernel address leakages in normal data flows that reveal the kernel address clues. Hawkeye precisely identifies the kernel addresses with minimum manual annotation and scales to analyze the whole kernel source code. It requires nearly ten times fewer memory resources and 40 times less inspection time than the state-of-the-art tool that analyzes kernel address leakage. Hawkeye unveils hundreds of leakages in various versioned kernels. It has even discovered 20 bugs in the mainline Linux kernel with the kernel pointer hashing mechanism already deployed and three bugs in FreeBSD. All the corresponding patches have been accepted by the developers.

IPC (Inter-Process Communication) is a widely used operating system (OS) technique that allows one process to invoke the services of other processes. The IPC participants may share the same OS ( internal IPC ) or use a separate OS ( external IPC ). Even though a long line of researches has optimized the performance of IPC, it is still a major factor of the run-time overhead of IPC-intensive applications. Furthermore, there is no one-size-fits-all solution for both internal and external IPC. This paper presents SkyBridge, a general communication technique designed and optimized for both types of IPC. SkyBridge requires no involvement of the privileged software (the kernel or the hypervisor) and enables a process to directly switch to the virtual address space of the target process, regardless of whether they are running on the same OS or not. We have implemented SkyBridge on two microkernels (seL4 and Google Zircon) as well as an open-source serverless hypervisor (Firecracker). The evaluation results show that SkyBridge improves the latency of internal IPC and external IPC by up to 19.6x and 1265.7x, respectively.