RSRui Shi

Rui Shi

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Rui Shi has a total of 90 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    FLEDA: Forecasting-Based Data Partition Placement for Efficient Deep Learning Recommendation Model Training

    by Yuchen Yang, Teng Yin, Yang Guo, Qingheng Gao, Jixi Shan, Hang Cheng, Xiuqi Huang, Jun Song, Rui Shi, Wei Chen

    In recommendation and advertising systems, model training pipelines rely on high-throughput ingestion of fresh, collected behavioral data. These data are typically organized into time-partitioned, multi-level table paths. The access patterns driven by highly concurrent model training tasks exhibit complex cross-path correlations, multi-scale temporal dependencies, and frequent hot-cold transitions, making effective data placement under limited SSD capacity challenging. In this paper, we propose FLEDA, a forecasting-based data partition placement framework that selectively places daily-level training data partitions onto SSDs under constrained storage capacity. We first model historical data access patterns as multivariate and multiscale time series to capture both temporal dynamics and cross-path dependencies. By leveraging dependencies across two dimensions, we propose a model that extracts multi-scale cross-path temporal features and fuses mixture-of-experts to forecast the future throughput of all table paths. Besides, we design a ranking-based placement policy that balances predicted throughput, recent access history, and SSD residency continuity to maximize coverage of future data accesses while avoiding excessive data migration. Experiments on large-scale, real-world training workloads show that FLEDA consistently outperforms state-of-the-art multivariate time-series forecasting models and significantly improves SSDs' throughput coverage with low data migration overhead.

  • Open Access

    CLAPS: A Load-Aware Proxy Resource Pooling System for Reducing Resource Redundancy in Large-Scale Cloud Storage

    by Xiuqi Huang, Shiwei Cao, Honghao Wang, Ruijie He, Huameng Zhou, Cheng Wang, Linyu Chen, Qingsong Zhu, Shunxin Min, Yunlong Wang, et al.

  • Open Access

    FoundRoot: Towards Foundation Model for Root Cause Analysis via Structured Deep Thinking

    by Zhe Xie, Zeyan Li, Xiao He, Shenglin Zhang, Longlong Xu, Yuzhuo Yang, Tieying Zhang, Jianjun Chen, Rui Shi, Dan Pei


  • Open Access

    Coarse-Fine: A Novel VPA Strategy for Boosting Resource Utilization of Transient Offline Tasks

    by Yingying Wen, Yingzhi Chu, Zhongyu Wang, Zhecheng Lin, Fan Jiang, Wu Xiang, Rui Shi

  • Open Access

    AgentTune: An Agent-Based Large Language Model Framework for Database Knob Tuning

    by Yiyan Li, Haoyang Li, Jing Zhang, Renata Borovica-Gajic, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Cuiping Li, Hong Chen

    Database knob tuning is a long-standing challenge in the database community, aimed at enhancing the performance of database management systems (DBMSs) by minimizing latency and maximizing throughput. Manual tuning, which relies heavily on human expertise, is often inefficient and impractical for large-scale or dynamic deployments. Recent work has explored automating this process using machine learning (ML) and large language models (LLMs). However, existing methods typically require hundreds of workload replays or rely on extensive training data, leading to low tuning efficiency or high preparation costs. Moreover, they also risk generating invalid configurations that can degrade performance or even crash the database. To address these limitations, we introduce AgentTune, the first agent-based knob tuning framework powered by LLMs, designed for efficiency, adaptability, and reliability. AgentTune decomposes the tuning process into four specialized agents: Workload Analyzer, Knob Selector, Range Pruner, and Configuration Recommender, each responsible for a distinct subtask. These agents collaborate through structured prompt chaining. AgentTune first analyzes the input workload to identify impactful knobs and reconstructs their valid ranges to reduce the search space. It then employs a tree-based search strategy to efficiently explore the configuration space and identify suitable knob values. We conduct extensive evaluations across diverse workloads (public benchmarks and real-world workloads), metrics (latency and throughput), DBMSs (PostgreSQL, MySQL, and TiDB), hardware environments, and database scales. Experimental results demonstrate that, compared to existing baselines, AgentTune is able to identify superior configurations using significantly fewer workload replays. Furthermore, AgentTune rarely generates invalid configurations during the tuning process, ensuring reliability and suitability for real-world deployments.


  • Open Access

    From Bottleneck to Breakthrough: Optimizing Scheduling for Hyperscale Containerized Clusters

    by Bing Li, Yuquan Ren, Xinyi Song, Zhilei Liu, Cong Xu, Jingyuan Zhang, Caixue Lin, Wu Xiang, Rui Shi


  • Open Access

    TickIt: Leveraging Large Language Models for Automated Ticket Escalation

    by Fengrui Liu, Xiao He, Tieying Zhang, Jianjun Chen, Yi Li, Lihua Yi, Haipeng Zhang, Gang Wu, Rui Shi


  • Open Access

    PLM4NDV: Minimizing Data Access for Number of Distinct Values Estimation with Pre-trained Language Models

    by Xianghong Xu, Xiao He, Tieying Zhang, Lei Zhang, Rui Shi, Jianjun Chen

    Number of Distinct Values (NDV) estimation of a multiset/column is a basis for many data management tasks, especially within databases. Despite decades of research, most existing methods require either a significant amount of samples through uniform random sampling or access to the entire column to produce estimates, leading to substantial data access costs and potentially ineffective estimations in scenarios with limited data access. In this paper, we propose leveraging semantic information, i.e., schema, to address these challenges. The schema contains rich semantic information that can benefit the NDV estimation. To this end, we propose PLM4NDV, a learned method incorporating Pre-trained Language Models (PLMs) to extract semantic schema information for NDV estimation. Specifically, PLM4NDV leverages the semantics of the target column and the corresponding table to gain a comprehensive understanding of the column's meaning. By using the semantics, PLM4NDV reduces data access costs, provides accurate NDV estimation, and can even operate effectively without any data access. Extensive experiments on a large-scale real-world dataset demonstrate the superiority of PLM4NDV over baseline methods. Our code is available at https://github.com/bytedance/plm4ndv.


  • Open Access

    ByteMQ: A Cloud-native Streaming Data Layer in ByteDance

    by Yancan Mao, Ruohang Yin, Liyuan Lei, Peng Ye, Shengfu Zou, Shizheng Tang, Yunzhe Guo, Ye Yuan, Xiaochen Yu, Bo Wan, et al.

    Real-time streaming data is generated in high volumes and consumed for statistical and analytical purposes, requiring efficient and effective management by Message Queuing Systems (MQS) that ensure high throughput and low latency. ByteDance relies extensively on MQS to handle its massive streaming data across various applications. However, existing MQS solutions often fall short of meeting ByteDance's high-volume, diverse requirements. To address these challenges, we propose ByteMQ (BMQ), a cloud-native streaming data layer designed to manage ByteDance's extensive streaming data needs efficiently in the cloud. BMQ features three key designs: 1) separation of messaging and storage, utilizing ByteDance's Federated Distributed File System (DFS) for high-performance data storage; 2) adaptive resource scheduling to balance workloads and redistribute resources across multiple availability zones; and 3) historical data restructuring to support offline applications with efficient structured data management. ByteDance has migrated 99.76% of its Kafka clusters to BMQ infrastructure, achieving about a 70% reduction in resource costs. This paper shares our journey of designing and implementing BMQ, providing insights that may benefit other organizations facing similar challenges.


  • Open Access

    Gödel

    by Wu Xiang, Yakun Li, Yuquan Ren, Fan Jiang, Chaohui Xin, Varun Gupta, Chao Xiang, Xinyi Song, Meng Liu, Bing Li, et al.

    Over the last few years, at ByteDance, our compute infrastructure scale has been expanding significantly due to expedited business growth. In this journey, to meet hyper-scale growth, some business groups resorted to managing their own compute infrastructure stack running different scheduling systems such as Kubernetes, YARN which created two major pain points: the increasing resource fragmentation across different business groups and the inadequate resource elasticity between workloads of different business priorities. Isolation across different business groups (and their compute infrastructure management) leads to inefficient compute resource utilization and prevents us from serving the business growth needs in the long run. To meet these challenges, we propose a resource management and scheduling system named Gödel, which provides a unified compute infrastructure for all business groups to run their diverse workloads under a unified resource pool. It co-locates various workloads on every machine to achieve better resource utilization and elasticity. Gödel is built upon Kubernetes, the de facto open-source container orchestration system, but with significant components replaced or enhanced to accommodate various workloads at a large scale. In production, it manages clusters with tens of thousands of machines, achieves high overall resource utilization of over 60%, and scheduling throughput of up to 5000 pods per second. This paper reports on our design and implementation with Gödel. Moreover, it discusses the lessons and best practices we learned in developing and operating it in production at ByteDance's scale.