Zero+: Monitoring Large-Scale Cloud-Native Infrastructure Using One-Sided RDMA
- 27 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Networking
- Vol. 32 (4) , 3499-3514
Abstract
Cloud services have shifted from monolithic designs to microservices running on cloud-native infrastructure with monitoring systems to ensure service level agreements (SLAs). However, traditional monitoring systems no longer meet the demands of cloud-native monitoring. In Alibaba’s “double eleven” shopping festival, it is observed that the monitor occupies resources of the monitored infrastructure and even disrupts services. In this paper, we propose a novel monitoring system named for cloud-native monitoring. achieves zero overhead in collecting raw metrics using one-sided remote direct memory access (RDMA) and remedies network congestion by adopting a receiver-driven flow control scheme. also features a priority queue mechanism to meet different quality of service requirements and an efficient batch processing design to relieve CPU occupation. has been deployed and evaluated in four different clusters with heterogeneous RDMA NIC devices and architectures in Alibaba Cloud. Results show that achieves no CPU occupation at the monitored host and supports $1\sim10k$ hosts with $0.1\sim1s$ sampling interval using a single thread for network I/O. significantly relieves the incast issue and maintains $80\sim95\%$ of bandwidth utilization in several clusters when monitoring $1k$ hosts. also ensures services with high priority accomplish collecting metrics earlier than low priority ones by at least $400 \mu s$ when monitoring $1k$ hosts.Keywords
Funding Information
- National Natural Science Foundation of China (62141220, 61972253, U1908212, 72061127001, 62172276, 61972254)
- Program for Professor of Special Appointment at Shanghai Institutions of Higher Learning
This publication has 41 references indexed in Scilit:
- Accelerated Information Dissemination for Replica Selection in Distributed Key-Value Store SystemsIEEE Transactions on Parallel and Distributed Systems, 2022
- Liquid: Intelligent Resource Estimation and Network-Efficient Scheduling for Deep Learning Jobs on Distributed GPU ClustersIEEE Transactions on Parallel and Distributed Systems, 2021
- Adaptive Resource Efficient Microservice Deployment in Cloud-Edge ContinuumIEEE Transactions on Parallel and Distributed Systems, 2021
- Thinking More about RDMA Memory Semantics2021
- Characterizing Co-Located Workloads in Alibaba Cloud DatacentersIEEE Transactions on Cloud Computing, 2020
- X-RDMA: Effective RDMA Middleware in Large-scale Production Environments2019
- RF-RPC: Remote Fetching RPC Paradigm for RDMA-Enabled NetworkIEEE Transactions on Parallel and Distributed Systems, 2018
- Visualization of Health Monitoring Data Acquired from Distributed Sensors for Multiple Patients2015
- Cloud monitoring: A surveyComputer Networks, 2013
- The ganglia distributed monitoring system: design, implementation, and experienceParallel Computing, 2004