Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13451-13462
Abstract
Perhaps surprisingly sewerage infrastructure is one of the most costly infrastructures in modern society. Sewer pipes are manually inspected to determine whether the pipes are defective. However, this process is limited by the number of qualified inspectors and the time it takes to inspect a pipe. Automatization of this process is therefore of high interest. So far, the success of computer vision approaches for sewer defect classification has been limited when compared to the success in other fields mainly due to the lack of public datasets. To this end, in this work we present a large novel and publicly available multi-label classification dataset for image-based sewer defect classification called Sewer-ML.The Sewer-ML dataset consists of 1.3 million images annotated by professional sewer inspectors from three different utility companies across nine years. Together with the dataset, we also present a benchmark algorithm and a novel metric for assessing performance. The benchmark algorithm is a result of evaluating 12 state-of-the-art algorithms, six from the sewer defect classification domain and six from the multi-label classification domain, and combining the best performing algorithms. The novel metric is a class-importance weighted F2 score, F2 CIW , reflecting the economic impact of each class, used together with the normal pipe F1 score, F1 Normal . The benchmark algorithm achieves an F2 CIW score of 55.11% and F1 Normal score of 90.94%, leaving ample room for improvement on the Sewer-ML dataset. The code, models, and dataset are available at the project page http://vap.aau.dk/sewer-mlKeywords
All Related Versions
Funding Information
- Innovation Fund
This publication has 74 references indexed in Scilit:
- Defect Detection in Pipes using a Mobile Laser-Optics Technology and Digital GeometryMATEC Web of Conferences, 2015
- HCP: A Flexible CNN Framework for Multi-Label Image ClassificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- State of the art review of inspection technologies for condition assessment of water pipesMeasurement, 2013
- Evaluation of ultrasonic inspection and imaging systems for concrete pipesAutomation in Construction, 2012
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- NUS-WIDE2009
- Extremely randomized treesMachine Learning, 2006
- State of the art in sensor technologies for sewer inspectionIEEE Sensors Journal, 2002
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial EnvelopeInternational Journal of Computer Vision, 2001