ExaQuery: Proving Data Structure to Unstructured Telemetry Data in Large-Scale HPC
Open Access
- 7 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 127-134
Abstract
High-performance computing (HPC) is the cornerstone of technological advancements in our digital age, but its management is becoming increasingly challenging, particularly as systems approach exascale. Operational data analytics (ODA) and holistic monitoring frameworks aim to alleviate this burden by collecting live telemetry from HPC systems. ODA frameworks rely on NoSQL databases for scalability, with implicit data structures embedded in metric names, necessitating domain knowledge for navigating telemetry data relations. To address the imperative need for explicit representation of relations in telemetry data, we propose a novel ontology for ODA, which we apply to a real HPC installation. The proposed ontology captures relationships between topological components and links hardware components(compute nodes, rack, systems) with job's execution and allocations collected telemetry. This ontology forms the basis for constructing a knowledge graph, enabling graph queries for ODA. Moreover, we propose a comparative analysis of the complexity (expressed in lines of code) and domain knowledge requirement (qualitatively assessed by informed end-users) of complex query implementation with the proposed method and NoSQL methods commonly employed in today's ODAs. We focused on six queries informed by facility managers' daily operations, aiming to benefit not only facility managers but also system administrators and user support. Our comparative analysis demonstrates that the proposed ontology facilitates the implementation of complex queries with significantly fewer lines of code and domain knowledge required as compared to NoSQL methods.Keywords
Funding Information
- HE EU Graph-Massivizer project (101093202)
- EuroHPC EU Regale project (956560)
- Spoke FutureHPC & BigData of the ICSC ? Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing, funded by European Union ? NextGenerationEU
This publication has 10 references indexed in Scilit:
- Democratizing HPC Access and Use with Knowledge Graphs2023
- Graph Neural Networks for Anomaly Anticipation in HPC Systems2023
- A survey of RDF stores & SPARQL engines for querying knowledge graphsThe VLDB Journal, 2021
- Revealing power, energy and thermal dynamics of a 200PF pre-exascale supercomputer2021
- HPC Ontology: Towards a Unified Ontology for Managing Training Datasets and AI Models for High-Performance Computing2021
- A High-Level Ontology Network for ICT Infrastructures2021
- Ontology-based Framework for Integration of Time Series Data: Application in Predictive Analytics on Data Center Monitoring Metrics2021
- Global Experiences with HPC Operational Data Measurement, Collection and Analysis2020
- Paving the Way Toward Energy-Aware and Automated Datacentre2019
- An ontology for heterogeneous resources management interoperability and HPC in the cloudFuture Generation Computer Systems, 2018