Spatio-temporal factorization of log data for understanding network events
Authors:
- 1 April 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 610-618
Abstract
Understanding the impacts and patterns of network events such as link flaps or hardware errors is crucial for diagnosing network anomalies. In large production networks, analyzing the log messages that record network events has become a challenging task due to the following two reasons. First, the log messages are composed of unstructured text messages generated by vendor-specific rules. Second, network equipment such as routers, switches, and RADIUS severs generate various log messages induced by network events that span across several geographical locations, network layers, protocols, and services. In this paper, we have tackled these obstacles by building two novel techniques: statistical template extraction (STE) and log tensor factorization (LTF). STE leverages a statistical clustering technique to automatically extract primary templates from unstructured log messages. LTF aims to build a statistical model that captures spatial-temporal patterns of log messages. Such spatial-temporal patterns provide useful insights into understanding the impacts and root cause of hidden network events. This paper first formulates our problem in a mathematical way. We then validate our techniques using massive amount of network log messages collected from a large operating network. We also demonstrate several case studies that validate the usefulness of our technique.Keywords
This publication has 16 references indexed in Scilit:
- Rapid detection of maintenance induced changes in service performance2011
- Spatio-temporal patterns in network events2010
- What happened in my network2010
- Detecting the performance impact of upgrades in large operational networksACM SIGCOMM Computer Communication Review, 2010
- Fault Localization via Risk ModelingIEEE Transactions on Dependable and Secure Computing, 2009
- Detailed diagnosis in enterprise networks2009
- Learning, indexing, and diagnosing network faults2009
- What's going on?2008
- Troubleshooting chronic conditions in large IP networks2008
- Learning the parts of objects by non-negative matrix factorizationNature, 1999