Jing Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Jing Liu has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Feiran Hu
Cloud Database Innovation Lab of Cloud BU, Huawei Technologies Co., Chengdu, China
1 shared publication - Guanglin Cong
Cloud Database Innovation Lab of Cloud BU, Huawei Technologies Co., Chengdu, China
1 shared publication - Jihan Qu
University of Electronic Science and Technology of China, Chengdu, China
1 shared publication - Kai Zheng
University of Electronic Science and Technology of China
1 shared publication - Ran Xu
Cloud Database Innovation Lab of Cloud BU, Huawei Technologies Co., Chengdu, China
1 shared publication - Weiqi Li
Cloud Database Innovation Lab of Cloud BU, Huawei Technologies Co., Chengdu, China
1 shared publication
Latest Publications
- Open Access
- 9 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 201-213
ESTELLE: An Efficient and Cost-effective Cloud Log Engine
With the advancement of cloud computing, more and more enterprises are adopting cloud services to build a variety of applications. Monitoring and observability are integral to the complex and fragile cloud-native architecture. As an extremely important data source for both, logs play an indispensable role in applications such as code debugging, root cause analysis, troubleshooting, and trend analysis. However, the inherent characteristic of cloud logs, with TB-level daily data production per user and continuous growth over time and with business, poses core challenges for log engines. Traditional log management systems are inadequate for handling the requirements of massive log data high-frequency writing and storage, along with low-frequency retrieval and analysis in cloud environments. Exploring a low-cost, high-performance cloud-native log engine solution is an extremely extraordinary challenging task. To tackle these challenges, we propose a cost-effective cloud-native log engine, called ESTELLE, equipped with a low-cost pluggable log index framework. This engine features a compute-storage separation and read-write separation architecture, enabling linear scalability. We designed a near-lock-free writing process for handling high-frequency writing demands of massive logs. Object storage is used to significantly reduce storage costs. We also tailored ESTELLE Log Bloom filter and approximate inverted index for this cloud-native engine, applying them flexibly to enhance query efficiency and optimize various queries. Extensive experiments on real open-source log datasets have demonstrated that the ESTELLE Log Engine achieves ultra-high single-core CPU write speeds and pretty low storage costs. Furthermore, when equipped with the complete index framework, it also maintains fairly low query latency across various log scenarios.
- 9 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 201-213
Open AccessESTELLE: An Efficient and Cost-effective Cloud Log Engine
With the advancement of cloud computing, more and more enterprises are adopting cloud services to build a variety of applications. Monitoring and observability are integral to the complex and fragile cloud-native architecture. As an extremely important data source for both, logs play an indispensable role in applications such as code debugging, root cause analysis, troubleshooting, and trend analysis. However, the inherent characteristic of cloud logs, with TB-level daily data production per user and continuous growth over time and with business, poses core challenges for log engines. Traditional log management systems are inadequate for handling the requirements of massive log data high-frequency writing and storage, along with low-frequency retrieval and analysis in cloud environments. Exploring a low-cost, high-performance cloud-native log engine solution is an extremely extraordinary challenging task. To tackle these challenges, we propose a cost-effective cloud-native log engine, called ESTELLE, equipped with a low-cost pluggable log index framework. This engine features a compute-storage separation and read-write separation architecture, enabling linear scalability. We designed a near-lock-free writing process for handling high-frequency writing demands of massive logs. Object storage is used to significantly reduce storage costs. We also tailored ESTELLE Log Bloom filter and approximate inverted index for this cloud-native engine, applying them flexibly to enhance query efficiency and optimize various queries. Extensive experiments on real open-source log datasets have demonstrated that the ESTELLE Log Engine achieves ultra-high single-core CPU write speeds and pretty low storage costs. Furthermore, when equipped with the complete index framework, it also maintains fairly low query latency across various log scenarios.