CloudRaid: Detecting Distributed Concurrency Bugs via Log Mining and Enhancement
- 2 June 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 48 (2) , 662-677
Abstract
Cloud systems suffer from distributed concurrency bugs, which often lead to data loss and service outage. This paper presents CLOUDRAID, a new automatical tool for finding distributed concurrency bugs efficiently and effectively. Distributed concurrency bugs are notoriously difficult to find as they are triggered by untimely interaction among nodes, i.e., unexpected message orderings. To detect concurrency bugs in cloud systems efficiently and effectively, CLOUDRAID analyzes and tests automatically only the message orderings that are likely to expose errors. Specifically, CLOUDRAID mines the logs from previous executions to uncover the message orderings that are feasible but inadequately tested. In addition, we also propose a log enhancing technique to introduce new logs automatically in the system being tested. These extra logs added improve further the effectiveness of CLOUDRAID without introducing any noticeable performance overhead. Our log-based approach makes it well-suited for live systems. We have applied CLOUDRAID to analyze six representative distributed systems: Apache Hadoop2/Yarn, HBase, HDFS, Cassandra, Zookeeper, and Flink. CLOUDRAID has succeeded in testing 60 different versions of these six systems (10 versions per system) in 35 hours, uncovering 31 concurrency bugs, including nine new bugs that have never been reported before. For these nine new bugs detected, which have all been confirmed by their original developers, three are critical and have already been fixed.Keywords
Funding Information
- National Key Research and Development Program of China (2016YFB1000201)
- National Natural Science Foundation of China (61802368, 61521092, 61432016, 61432018, 61332009, 61702485, 61872043)
- CCF-Tencent Open Research Fund
- Australian Research Council (DP170103956, DP180104069)
This publication has 40 references indexed in Scilit:
- Race detection for event-driven mobile applicationsACM SIGPLAN Notices, 2014
- Improving Software Diagnosability via Log EnhancementACM Transactions on Computer Systems, 2012
- Mining temporal invariants from partially ordered logsACM SIGOPS Operating Systems Review, 2012
- ColorSafeACM SIGARCH Computer Architecture News, 2010
- CassandraACM SIGOPS Operating Systems Review, 2010
- Mining dependency in distributed systems through unstructured logs analysisACM SIGOPS Operating Systems Review, 2010
- FastTrackACM SIGPLAN Notices, 2009
- MapReduceCommunications of the ACM, 2008
- AVIOACM SIGOPS Operating Systems Review, 2006
- EraserACM Transactions on Computer Systems, 1997