AIOpsArena: Scenario-Oriented Evaluation and Leaderboard for AIOps Algorithms in Microservices
Authors:
- 4 March 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 809-813
Abstract
AIOps algorithms playa crucial role in the mainte-nance of microservice systems. Many previous benchmarks' per-formance leaderboard provides valuable guidance for selecting appropriate algorithms. However, existing AIOps benchmarks mainly utilize offline static datasets to evaluate algorithms. They cannot consistently evaluate the performance of algorithms using real-time datasets, and the operation scenarios for evaluation are static, which is insufficient for effective algorithm selection. To address these issues, we propose an evaluation-consistent and scenario-oriented evaluation framework named AIOpsArena. The core idea is to build a live microservice benchmark to generate real-time datasets and consistently simulate the specific operation scenarios on it. AIOpsArena supports different leaderboards by selecting specific algorithms and datasets according to the operation scenarios. It also supports the deployment of various types of algorithms, enabling algorithms hot-plugging. At last, we test AIOpsArena with typical microservice operation scenarios to demonstrate its efficiency and usability. Platform and a video demonstrating the functioning of AIOpsArena is available from https://github.com/AIOpsArena/aiopsarena.Keywords
This publication has 10 references indexed in Scilit:
- TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Models2024
- MicroOps: Rapid Microservice Data Simulation and AIOps Model Development Platform2024
- Are Transformers Effective for Time Series Forecasting?Proceedings of the AAAI Conference on Artificial Intelligence, 2023
- An automatic model management system and its implementation for AIOps on microservice platformsThe Journal of Supercomputing, 2023
- VECROsim: A Versatile Metric-oriented Microservice Fault Simulation System (Tools and Artifact Track)2022
- MicroCBR: Case-Based Reasoning on Spatio-temporal Fault Knowledge Graph for Microservices Troubleshooting2022
- Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A SurveyACM Computing Surveys, 2022
- USAD2020
- DeepLog2017
- A Systematic Mapping Study in Microservice Architecture2016