Practical DB-OS Co-Design with Privileged Kernel Bypass
Open Access
- 10 February 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 3 (1) , 1-27
Abstract
This paper revisits the longstanding challenge of coordinating database systems with general-purpose OS interfaces, such as POSIX, which often lack tailored support for DB requirements. Existing approaches to this DB-OS co-design struggle with limited design space, security risks, and compatibility issues. To overcome these hurdles, we propose a new co-design approach leveraging virtualization to elevate the privilege level of DB processes. Our method enables database systems to fully exploit hardware capabilities via virtualization, while minimizing the need for extensive modifications to the host OS kernel, thereby maintaining compatibility. We demonstrate the effectiveness of our approach through two novel virtual memory mechanisms tailored for database workloads: (1) an efficient snapshotting mechanism that captures memory snapshots at millisecond intervals for in-memory databases and HTAP workloads, and (2) a streamlined in-kernel buffer pool design. We introduce Libdbos , a lightweight guest kernel implementing these mechanisms. Our evaluations highlight significant improvements in latency and efficiency compared to existing snapshotting and buffer pool designs, underscoring the potential of the approach.Keywords
This publication has 37 references indexed in Scilit:
- A Three-Tier Buffer Manager Integrating CXL Device Memory for Database Systems2024
- Packed Memory Arrays - Rewired2019
- SPDK: A Development Kit to Build High Performance Storage Applications2017
- The tail at scaleCommunications of the ACM, 2013
- HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots2011
- The effects of virtualization on main memory systems2010
- Benchmarking Database Performance in a Virtual Environment2009
- Database systems on virtual machines: How much do you lose?2008
- Database Virtualization: A New Frontier for Database Tuning and Physical Design2007
- Quickly generating billion-record synthetic databases1994