WebGraph: The Next Generation (Is in Rust)

Authors:
Open Access
Publisher website
Google Scholar
Abstract
We report the results of a yearlong effort at the Laboratory for Web Algorithmics and Inria to port the WebGraph framework [4] from Java to Rust. For two decades WebGraph has been instrumental in the analysis and distribution of large graphs for the research community of TheWebConf, but the intrinsic limitations of the Java Virtual Machine had become a bottleneck for very large use cases, such as the Software Heritage Merkle graph [2] with its half a trillion arcs. As part of this clean-slate implementation of WebGraph in Rust, we developed a few ancillary projects bringing to the Rust ecosystem some missing features of independent interest, such as easy, consistent and zero-cost memory mapping of data structures. WebGraph in Rust offers impressive performance improvements over the previous implementation, enabling open-source graph analytics on very large datasets like Common Crawl, on top of a modern systems programming language.
Funding Information
  • Ministero dell'Università e della Ricerca (SERICS (PE00000014))
  • Agencie National de la Recherche (COREGRAPHIE (ANR-20-CE23-0002))