Zengyin Yang
Zengyin Yang has a total of 26 co-authors. Here are the top co-authors by number of shared publications:
- Michael R. Lyu
The Chinese University of Hong Kong
4 shared publications - Cong Feng
Huawei Cloud Computing Technology Co., Ltd, Shenzhen, China
3 shared publications Jianping Wu
Tsinghua Univ, Dept Comp Sci & Technol, Beijing 100084, Peoples R China
3 shared publications- Jiazhen Gu
The Chinese University of Hong Kong, Hong Kong, China
3 shared publications - Jinyang Liu
Chinese University of Hong Kong, Hong Kong, China
3 shared publications - Junjie Huang
Chinese University of Hong Kong, Hong Kong, China
3 shared publications
Latest Publications
- Open Access
- 23 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 51-63
L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis
- 23 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 51-63
Open AccessL4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis
- Open Access
- 23 February 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 34 (3) , 1-31
Identifying Performance Issues in Cloud Service Systems Based on Relational-Temporal Features
Cloud systems, typically comprised of various components (e.g., microservices), are susceptible to performance issues, which may cause service-level agreement violations and financial losses. Identifying performance issues is thus of paramount importance for cloud vendors. In current practice, crucial metrics, i.e., Key Performance Indicators (KPIs), are monitored periodically to provide insight into the operational status of components. Identifying performance issues is often formulated as an anomaly detection problem, which is tackled by analyzing each metric independently. However, this approach overlooks the complex dependencies existing among cloud components. Some graph neural network-based methods take both temporal and relational information into account; however, the correlation violations in the metrics that serve as indicators of underlying performance issues are difficult for them to identify. Furthermore, a large volume of components in a cloud system results in a vast array of noisy metrics. This complexity renders it impractical for engineers to fully comprehend the correlations, making it challenging to identify performance issues accurately. To address these limitations, we propose Identifying Performance Issues based on Relational-Temporal Features (ISOLATE), a learning-based approach that leverages both the relational and temporal features of metrics to identify performance issues. In particular, it adopts a graph neural network with attention to characterizing the relations among metrics and extracts long-term and multi-scale temporal patterns using a GRU and a convolution network, respectively. The learned graph attention weights can be further used to localize the correlation-violated metrics. Moreover, to relieve the impact of noisy data, ISOLATE utilizes a Positive Unlabeled (PU) Learning strategy that tags pseudo-labels based on a small portion of confirmed negative examples. Extensive evaluation on both public and industrial datasets shows that ISOLATE outperforms all baseline models with 0.945 F1 score and 0.920 Hit rate@3. The ablation study also proves the effectiveness of the relational-temporal features and the PU-Learning strategy. Furthermore, we share the success stories of leveraging ISOLATE to identify performance issues in Huawei Cloud, which demonstrates its superiority in practice.
- 23 February 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 34 (3) , 1-31
Open AccessIdentifying Performance Issues in Cloud Service Systems Based on Relational-Temporal Features
Cloud systems, typically comprised of various components (e.g., microservices), are susceptible to performance issues, which may cause service-level agreement violations and financial losses. Identifying performance issues is thus of paramount importance for cloud vendors. In current practice, crucial metrics, i.e., Key Performance Indicators (KPIs), are monitored periodically to provide insight into the operational status of components. Identifying performance issues is often formulated as an anomaly detection problem, which is tackled by analyzing each metric independently. However, this approach overlooks the complex dependencies existing among cloud components. Some graph neural network-based methods take both temporal and relational information into account; however, the correlation violations in the metrics that serve as indicators of underlying performance issues are difficult for them to identify. Furthermore, a large volume of components in a cloud system results in a vast array of noisy metrics. This complexity renders it impractical for engineers to fully comprehend the correlations, making it challenging to identify performance issues accurately. To address these limitations, we propose Identifying Performance Issues based on Relational-Temporal Features (ISOLATE), a learning-based approach that leverages both the relational and temporal features of metrics to identify performance issues. In particular, it adopts a graph neural network with attention to characterizing the relations among metrics and extracts long-term and multi-scale temporal patterns using a GRU and a convolution network, respectively. The learned graph attention weights can be further used to localize the correlation-violated metrics. Moreover, to relieve the impact of noisy data, ISOLATE utilizes a Positive Unlabeled (PU) Learning strategy that tags pseudo-labels based on a small portion of confirmed negative examples. Extensive evaluation on both public and industrial datasets shows that ISOLATE outperforms all baseline models with 0.945 F1 score and 0.920 Hit rate@3. The ablation study also proves the effectiveness of the relational-temporal features and the PU-Learning strategy. Furthermore, we share the success stories of leveraging ISOLATE to identify performance issues in Huawei Cloud, which demonstrates its superiority in practice.
- Open Access
- 3 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 16-22
LubeRDMA: A Fail-safe Mechanism of RDMA
Recent years have witnessed a wide adoption of Remote Direct Memory Access (RDMA) to accelerate distributed systems. As the scale of distributed applications keeps increasing, network failures become more prominent. Although some link/switch failures can be circumvented by in-network rerouting, failures like NIC failure are still fatal in RDMA networks and may cause the entire system to fail. To address this issue, we propose a fail-safe mechanism of RDMA called LubeRDMA. The core idea is to leverage multiple RDMA NICs on a server and treat them as backups for each other. We introduce a vRDMA model that abstracts a failure-resilient RDMA network to the application. With this model, we achieve RDMA fault tolerance and recovery. In our evaluation, we demonstrate that LubeRDMA efficiently handles RDMA failures while having a minimal impact on RDMA performance.
- 3 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 16-22
Open AccessLubeRDMA: A Fail-safe Mechanism of RDMA
Recent years have witnessed a wide adoption of Remote Direct Memory Access (RDMA) to accelerate distributed systems. As the scale of distributed applications keeps increasing, network failures become more prominent. Although some link/switch failures can be circumvented by in-network rerouting, failures like NIC failure are still fatal in RDMA networks and may cause the entire system to fail. To address this issue, we propose a fail-safe mechanism of RDMA called LubeRDMA. The core idea is to leverage multiple RDMA NICs on a server and treat them as backups for each other. We introduce a vRDMA model that abstracts a failure-resilient RDMA network to the application. With this model, we achieve RDMA fault tolerance and recovery. In our evaluation, we demonstrate that LubeRDMA efficiently handles RDMA failures while having a minimal impact on RDMA performance.
- Open Access
- 14 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 392-404
FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems
Postmortem analysis is essential in the management of incidents within cloud systems, which provides valuable insights to improve system's reliability and robustness. At CloudA1, fault pattern profiling is performed during the postmortem phase, which involves the classification of incidents' faults into unique categories, referred to as fault pattern. By aggregating and analyzing these fault patterns, engineers can discern common faults, vulnerable components and emerging fault trends. However, this process is currently conducted by manual labeling, which has inherent drawbacks. On the one hand, the sheer volume of incidents means only the most severe ones are analyzed, causing a skewed overview of fault patterns. On the other hand, the complexity of the task demands extensive domain knowledge, which leads to errors and inconsistencies. To address these limitations, we propose an automated approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets. It leverages hierarchy-guided contrastive learning to train a hierarchy-aware incident encoder and predicts fault patterns with enhanced incident representations. We evaluate FaultProfIT using the production incidents from CloudA. The results demonstrate that Fault-ProfIT outperforms state-of-the-art methods. Our ablation study and analysis also verify the effectiveness of hierarchy-guided contrastive learning. Additionally, we have deployed FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+ incidents from 30+ cloud services, successfully revealing several fault trends that have informed system improvements.
- 14 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 392-404
Open AccessFaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems
Postmortem analysis is essential in the management of incidents within cloud systems, which provides valuable insights to improve system's reliability and robustness. At CloudA1, fault pattern profiling is performed during the postmortem phase, which involves the classification of incidents' faults into unique categories, referred to as fault pattern. By aggregating and analyzing these fault patterns, engineers can discern common faults, vulnerable components and emerging fault trends. However, this process is currently conducted by manual labeling, which has inherent drawbacks. On the one hand, the sheer volume of incidents means only the most severe ones are analyzed, causing a skewed overview of fault patterns. On the other hand, the complexity of the task demands extensive domain knowledge, which leads to errors and inconsistencies. To address these limitations, we propose an automated approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets. It leverages hierarchy-guided contrastive learning to train a hierarchy-aware incident encoder and predicts fault patterns with enhanced incident representations. We evaluate FaultProfIT using the production incidents from CloudA. The results demonstrate that Fault-ProfIT outperforms state-of-the-art methods. Our ablation study and analysis also verify the effectiveness of hierarchy-guided contrastive learning. Additionally, we have deployed FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+ incidents from 30+ cloud services, successfully revealing several fault trends that have informed system improvements.
- Open Access
- 14 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 369-380
Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach
Due to the scale and complexity of cloud systems, a system failure would trigger an "alert storm", i.e., massive correlated alerts. Although these alerts can be traced back to a few root causes, the overwhelming number makes it infeasible for manual handling. Alert aggregation is thus critical to help engineers concentrate on the root cause and facilitate failure resolution. Existing methods typically utilize semantic similarity-based methods or statistical methods to aggregate alerts. However, semantic similarity-based methods overlook the causal rationale of alerts, while statistical methods can hardly handle infrequent alerts. To tackle these limitations, we introduce leveraging external knowledge, i.e., Standard Operation Procedure (SOP) of alerts as a supplement. We propose COLA, a novel hybrid approach based on correlation mining and LLM (Large Language Model) reasoning for online alert aggregation. The correlation mining module effectively captures the temporal and spatial relations between alerts, measuring their correlations in an efficient manner. Subsequently, only uncertain pairs with low confidence are forwarded to the LLM reasoning module for detailed analysis. This hybrid design harnesses both statistical evidence for frequent alerts and the reasoning capabilities of computationally intensive LLMs, ensuring the overall efficiency of COLA in handling large volumes of alerts in practical scenarios. We evaluate COLA on three datasets collected from the production environment of a large-scale cloud platform. The experimental results show COLA achieves F1-scores from 0.901 to 0.930, outperforming state-of-the-art methods and achieving comparable efficiency. We also share our experience in deploying COLA in our real-world cloud system, Cloud X1.
- 14 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 369-380
Open AccessKnowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach
Due to the scale and complexity of cloud systems, a system failure would trigger an "alert storm", i.e., massive correlated alerts. Although these alerts can be traced back to a few root causes, the overwhelming number makes it infeasible for manual handling. Alert aggregation is thus critical to help engineers concentrate on the root cause and facilitate failure resolution. Existing methods typically utilize semantic similarity-based methods or statistical methods to aggregate alerts. However, semantic similarity-based methods overlook the causal rationale of alerts, while statistical methods can hardly handle infrequent alerts. To tackle these limitations, we introduce leveraging external knowledge, i.e., Standard Operation Procedure (SOP) of alerts as a supplement. We propose COLA, a novel hybrid approach based on correlation mining and LLM (Large Language Model) reasoning for online alert aggregation. The correlation mining module effectively captures the temporal and spatial relations between alerts, measuring their correlations in an efficient manner. Subsequently, only uncertain pairs with low confidence are forwarded to the LLM reasoning module for detailed analysis. This hybrid design harnesses both statistical evidence for frequent alerts and the reasoning capabilities of computationally intensive LLMs, ensuring the overall efficiency of COLA in handling large volumes of alerts in practical scenarios. We evaluate COLA on three datasets collected from the production environment of a large-scale cloud platform. The experimental results show COLA achieves F1-scores from 0.901 to 0.930, outperforming state-of-the-art methods and achieving comparable efficiency. We also share our experience in deploying COLA in our real-world cloud system, Cloud X1.
- Open Access
- 20 March 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 46831-46845
NPVT: Network Protocol Validation Testbed for Integrated Space-Terrestrial Network
- 20 March 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 46831-46845
Open AccessNPVT: Network Protocol Validation Testbed for Integrated Space-Terrestrial Network
- Open Access
- 11 February 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 44721-44733
CRO-SS: A Route Oscillation Suppression Scheme for IP-Based Satellite Network
With the increasing demand for Internet access in rural, remote, and other areas, using the IP-based satellite network has become a trend to provide global Internet services. An important aspect of building this network is to achieve an efficient routing. However, when we conducted a systematic analysis of the routing efficiency of such a network, we found a severe route oscillation problem in the cross region of satellite orbits. This problem occurs every 67.4 seconds and causes 8.3%~24.1% of end-to-end paths in the entire network to flap, thus seriously damaging the routing efficiency and network performance. We term this the cross route oscillation (CRO) problem and found that it is primarily caused by the conflict between the path length and path lifetime. Subsequently, we propose the cross route oscillation suppression scheme (CRO-SS) to suppress CRO by decoupling that conflict. Extensive experiments using real-world data indicate that CRO-SS can significantly improve routing stability and network performance. The occurrence frequency of route flapping, the total number of changed paths, and the total number of affected satellite nodes over a system period are approximately 24.7%, 58.5%, and 63.1%, respectively. Meanwhile, the throughput is increased by 20% and the packet loss rate is decreased by 72.4%~97.8%.
- 11 February 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 44721-44733
Open AccessCRO-SS: A Route Oscillation Suppression Scheme for IP-Based Satellite Network
With the increasing demand for Internet access in rural, remote, and other areas, using the IP-based satellite network has become a trend to provide global Internet services. An important aspect of building this network is to achieve an efficient routing. However, when we conducted a systematic analysis of the routing efficiency of such a network, we found a severe route oscillation problem in the cross region of satellite orbits. This problem occurs every 67.4 seconds and causes 8.3%~24.1% of end-to-end paths in the entire network to flap, thus seriously damaging the routing efficiency and network performance. We term this the cross route oscillation (CRO) problem and found that it is primarily caused by the conflict between the path length and path lifetime. Subsequently, we propose the cross route oscillation suppression scheme (CRO-SS) to suppress CRO by decoupling that conflict. Extensive experiments using real-world data indicate that CRO-SS can significantly improve routing stability and network performance. The occurrence frequency of route flapping, the total number of changed paths, and the total number of affected satellite nodes over a system period are approximately 24.7%, 58.5%, and 63.1%, respectively. Meanwhile, the throughput is increased by 20% and the packet loss rate is decreased by 72.4%~97.8%.
- 1 December 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Analyzing and optimizing BGP stability in future space-based internet
Future Space-Based Internet (FSBI) aims to provide global Internet access by interconnecting geosynchronous orbit (GEO), Medium Earth Orbit (MEO), Low Earth Orbit (LEO) satellites, and gateways on the ground. Border Gateway Protocol (BGP) is considered as a feasible routing protocol for interconnecting these independent facilities in FSBI. However, the vast Mobility-Related Topology Changes (MRTCs) in FSBI will put great stress on the BGP stability. We carry out a deep analysis on it and find that the routing updates of BGP will be triggered frequently. Many BGP routing updates will occur even before the accomplishment of previous updates and then the network will be instable for most of the time. To solve this problem, we build a Discrete-Time Topology Changes Aggregation (DT-TCA) scheme to improve BGP stability by making as many MRTCs as possible be triggered at the same time. Using a high-fidelity testbed and a virtualization-based emulator of FSBI, we show that DT-TCA dramatically improves the BGP stability in small-scale and large-scale scenarios.
- 1 December 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Analyzing and optimizing BGP stability in future space-based internet
Future Space-Based Internet (FSBI) aims to provide global Internet access by interconnecting geosynchronous orbit (GEO), Medium Earth Orbit (MEO), Low Earth Orbit (LEO) satellites, and gateways on the ground. Border Gateway Protocol (BGP) is considered as a feasible routing protocol for interconnecting these independent facilities in FSBI. However, the vast Mobility-Related Topology Changes (MRTCs) in FSBI will put great stress on the BGP stability. We carry out a deep analysis on it and find that the routing updates of BGP will be triggered frequently. Many BGP routing updates will occur even before the accomplishment of previous updates and then the network will be instable for most of the time. To solve this problem, we build a Discrete-Time Topology Changes Aggregation (DT-TCA) scheme to improve BGP stability by making as many MRTCs as possible be triggered at the same time. Using a high-fidelity testbed and a virtualization-based emulator of FSBI, we show that DT-TCA dramatically improves the BGP stability in small-scale and large-scale scenarios.