ZYZengyin Yang

Zengyin Yang

*This is an algorithmically generated author record based on publicly available information.

Zengyin Yang has a total of 26 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis

    by Zhihan Jiang, Junjie Huang, Guangba Yu, Zhuangbin Chen, Yichen Li, Renyi Zhong, Cong Feng, Yongqiang Yang, Zengyin Yang, Michael Lyu


  • Open Access

    Identifying Performance Issues in Cloud Service Systems Based on Relational-Temporal Features

    by Wenwei Gu, Jinyang Liu, Zhuangbin Chen, Jianping Zhang, Yuxin Su, Jiazhen Gu, Cong Feng, Zengyin Yang, Yongqiang Yang, Michael R. Lyu

    Cloud systems, typically comprised of various components (e.g., microservices), are susceptible to performance issues, which may cause service-level agreement violations and financial losses. Identifying performance issues is thus of paramount importance for cloud vendors. In current practice, crucial metrics, i.e., Key Performance Indicators (KPIs), are monitored periodically to provide insight into the operational status of components. Identifying performance issues is often formulated as an anomaly detection problem, which is tackled by analyzing each metric independently. However, this approach overlooks the complex dependencies existing among cloud components. Some graph neural network-based methods take both temporal and relational information into account; however, the correlation violations in the metrics that serve as indicators of underlying performance issues are difficult for them to identify. Furthermore, a large volume of components in a cloud system results in a vast array of noisy metrics. This complexity renders it impractical for engineers to fully comprehend the correlations, making it challenging to identify performance issues accurately. To address these limitations, we propose Identifying Performance Issues based on Relational-Temporal Features (ISOLATE), a learning-based approach that leverages both the relational and temporal features of metrics to identify performance issues. In particular, it adopts a graph neural network with attention to characterizing the relations among metrics and extracts long-term and multi-scale temporal patterns using a GRU and a convolution network, respectively. The learned graph attention weights can be further used to localize the correlation-violated metrics. Moreover, to relieve the impact of noisy data, ISOLATE utilizes a Positive Unlabeled (PU) Learning strategy that tags pseudo-labels based on a small portion of confirmed negative examples. Extensive evaluation on both public and industrial datasets shows that ISOLATE outperforms all baseline models with 0.945 F1 score and 0.920 Hit rate@3. The ablation study also proves the effectiveness of the relational-temporal features and the PU-Learning strategy. Furthermore, we share the success stories of leveraging ISOLATE to identify performance issues in Huawei Cloud, which demonstrates its superiority in practice.


  • Open Access

    LubeRDMA: A Fail-safe Mechanism of RDMA

    by Shengkai Lin, Qinwei Yang, Zengyin Yang, Yuchuan Wang, Shizhen Zhao

    Recent years have witnessed a wide adoption of Remote Direct Memory Access (RDMA) to accelerate distributed systems. As the scale of distributed applications keeps increasing, network failures become more prominent. Although some link/switch failures can be circumvented by in-network rerouting, failures like NIC failure are still fatal in RDMA networks and may cause the entire system to fail. To address this issue, we propose a fail-safe mechanism of RDMA called LubeRDMA. The core idea is to leverage multiple RDMA NICs on a server and treat them as backups for each other. We introduce a vRDMA model that abstracts a failure-resilient RDMA network to the application. With this model, we achieve RDMA fault tolerance and recovery. In our evaluation, we demonstrate that LubeRDMA efficiently handles RDMA failures while having a minimal impact on RDMA performance.


  • Open Access

    FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems

    by Junjie Huang, Jinyang Liu, Zhuangbin Chen, Zhihan Jiang, Yichen Li, Jiazhen Gu, Cong Feng, Zengyin Yang, Yongqiang Yang, Michael R. Lyu

    Postmortem analysis is essential in the management of incidents within cloud systems, which provides valuable insights to improve system's reliability and robustness. At CloudA1, fault pattern profiling is performed during the postmortem phase, which involves the classification of incidents' faults into unique categories, referred to as fault pattern. By aggregating and analyzing these fault patterns, engineers can discern common faults, vulnerable components and emerging fault trends. However, this process is currently conducted by manual labeling, which has inherent drawbacks. On the one hand, the sheer volume of incidents means only the most severe ones are analyzed, causing a skewed overview of fault patterns. On the other hand, the complexity of the task demands extensive domain knowledge, which leads to errors and inconsistencies. To address these limitations, we propose an automated approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets. It leverages hierarchy-guided contrastive learning to train a hierarchy-aware incident encoder and predicts fault patterns with enhanced incident representations. We evaluate FaultProfIT using the production incidents from CloudA. The results demonstrate that Fault-ProfIT outperforms state-of-the-art methods. Our ablation study and analysis also verify the effectiveness of hierarchy-guided contrastive learning. Additionally, we have deployed FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+ incidents from 30+ cloud services, successfully revealing several fault trends that have informed system improvements.


  • Open Access

    Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach

    by Jinxi Kuang, Jinyang Liu, Junjie Huang, Renyi Zhong, Jiazhen Gu, Lan Yu, Rui Tan, Zengyin Yang, Michael R. Lyu

    Due to the scale and complexity of cloud systems, a system failure would trigger an "alert storm", i.e., massive correlated alerts. Although these alerts can be traced back to a few root causes, the overwhelming number makes it infeasible for manual handling. Alert aggregation is thus critical to help engineers concentrate on the root cause and facilitate failure resolution. Existing methods typically utilize semantic similarity-based methods or statistical methods to aggregate alerts. However, semantic similarity-based methods overlook the causal rationale of alerts, while statistical methods can hardly handle infrequent alerts. To tackle these limitations, we introduce leveraging external knowledge, i.e., Standard Operation Procedure (SOP) of alerts as a supplement. We propose COLA, a novel hybrid approach based on correlation mining and LLM (Large Language Model) reasoning for online alert aggregation. The correlation mining module effectively captures the temporal and spatial relations between alerts, measuring their correlations in an efficient manner. Subsequently, only uncertain pairs with low confidence are forwarded to the LLM reasoning module for detailed analysis. This hybrid design harnesses both statistical evidence for frequent alerts and the reasoning capabilities of computationally intensive LLMs, ensuring the overall efficiency of COLA in handling large volumes of alerts in practical scenarios. We evaluate COLA on three datasets collected from the production environment of a large-scale cloud platform. The experimental results show COLA achieves F1-scores from 0.901 to 0.930, outperforming state-of-the-art methods and achieving comparable efficiency. We also share our experience in deploying COLA in our real-world cloud system, Cloud X1.


  • Open Access

    CRO-SS: A Route Oscillation Suppression Scheme for IP-Based Satellite Network

    by Zengyin Yang, Qian Wu, Hewu Li, Jianping Wu

    With the increasing demand for Internet access in rural, remote, and other areas, using the IP-based satellite network has become a trend to provide global Internet services. An important aspect of building this network is to achieve an efficient routing. However, when we conducted a systematic analysis of the routing efficiency of such a network, we found a severe route oscillation problem in the cross region of satellite orbits. This problem occurs every 67.4 seconds and causes 8.3%~24.1% of end-to-end paths in the entire network to flap, thus seriously damaging the routing efficiency and network performance. We term this the cross route oscillation (CRO) problem and found that it is primarily caused by the conflict between the path length and path lifetime. Subsequently, we propose the cross route oscillation suppression scheme (CRO-SS) to suppress CRO by decoupling that conflict. Extensive experiments using real-world data indicate that CRO-SS can significantly improve routing stability and network performance. The occurrence frequency of route flapping, the total number of changed paths, and the total number of affected satellite nodes over a system period are approximately 24.7%, 58.5%, and 63.1%, respectively. Meanwhile, the throughput is increased by 20% and the packet loss rate is decreased by 72.4%~97.8%.


  • Analyzing and optimizing BGP stability in future space-based internet

    by Zengyin Yang, Hewu Li, Qian Wu, Jianping Wu

    Future Space-Based Internet (FSBI) aims to provide global Internet access by interconnecting geosynchronous orbit (GEO), Medium Earth Orbit (MEO), Low Earth Orbit (LEO) satellites, and gateways on the ground. Border Gateway Protocol (BGP) is considered as a feasible routing protocol for interconnecting these independent facilities in FSBI. However, the vast Mobility-Related Topology Changes (MRTCs) in FSBI will put great stress on the BGP stability. We carry out a deep analysis on it and find that the routing updates of BGP will be triggered frequently. Many BGP routing updates will occur even before the accomplishment of previous updates and then the network will be instable for most of the time. To solve this problem, we build a Discrete-Time Topology Changes Aggregation (DT-TCA) scheme to improve BGP stability by making as many MRTCs as possible be triggered at the same time. Using a high-fidelity testbed and a virtualization-based emulator of FSBI, we show that DT-TCA dramatically improves the BGP stability in small-scale and large-scale scenarios.