DISTIL: Design and implementation of a scalable synchrophasor data processing system
Authors:
- 1 November 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 271-277
Abstract
The introduction and deployment of cheap, high precision, high-sample-rate next-generation synchrophasors en masse in both the transmission and distribution tier - while invaluable for event diagnosis, situational awareness and capacity planning - poses a problem for existing methods of phasor data analysis and storage. Addressing this, we present the design and implementation of a novel architecture for synchrophasor data analysis on distributed commodity hardware. At the core is a new feature-rich timeseries store, BTrDB. Capable of sustained writes and reads in excess of 16 million points per second per cluster node, advanced query functionality and highly efficient storage, this database enables novel analysis and visualization techniques. Leveraging this, a distillate framework has been developed that enables agile development of scalable analysis pipelines with strict guarantees on result integrity despite asynchronous changes in data or out of order arrival. Finally, the system is evaluated in a pilot deployment, archiving more than 216 billion raw datapoints and 515 billion derived datapoints from 13 devices in just 3.9TB. We show that the system is capable of scaling to handle complex analytics and storage for tens of thousands of next-generation synchrophasors on off-the-shelf servers.Keywords
This publication has 6 references indexed in Scilit:
- Addressing the challenges for integrating micro-synchrophasor data with operational system applications2014
- Scalability and Robustness of Time-Series Databases for Cloud-Native Monitoring of Industrial Processes2014
- M4Proceedings of the VLDB Endowment, 2014
- Micro-synchrophasors for distribution systems2014
- Compressing Phasor Measurement data2013
- RADOS2007