F3: The Open-Source Data File Format for the Future
Open Access
- 22 September 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 3 (4) , 1-27
Abstract
Columnar storage formats are the foundation for modern data analytics systems. The proliferation of open-source file formats (i.e., Parquet, ORC) allows seamless data sharing across disparate platforms. However, these formats were created over a decade ago for hardware and workload environments that are much different from today. Although these formats have incorporated some updates to their specification to adapt to these changes, not all deployments support those modifications, and too often systems cannot overcome the formats' deficiencies and limitations without a rewrite. In this paper, we present the F uture-proof File Format (F3) project. It is a next-generation open-source file format with interoperability, extensibility, and efficiency as its core design principles. F3 obviates the need to create a new format every time a shift occurs in data processing and computing by providing a data organization structure and a general-purpose API to allow developers to add new encoding schemes easily. Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native decoders are unavailable. To evaluate F3, we compared it against legacy and state-of-the-art open-source file formats. Our evaluations demonstrate the efficacy of F3's storage layout and the benefits of Wasm-driven decoding.Keywords
This publication has 24 references indexed in Scilit:
- Accelerating GPU Data Processing using FastLanes Compression2024
- LeCo: Lightweight Compression via Learning Serial CorrelationsProceedings of the ACM on Management of Data, 2024
- BtrBlocks: Efficient Columnar Compression for Data LakesProceedings of the ACM on Management of Data, 2023
- Order-Preserving Key Compression for In-Memory Search Trees2020
- SuRF2018
- Column Sketches2018
- Bringing the web up to speed with WebAssembly2017
- Column imprints2013
- All aboard the Databus!2012
- RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems2011