Learn from experience: Probabilistic prediction of perception performance to avoid failure
Authors:
- 8 October 2017
- journal article
- research article
- Published by SAGE Publications in The International Journal of Robotics Research
- Vol. 37 (9) , 981-995
Abstract
Despite significant advances in machine learning and perception over the past few decades, perception algorithms can still be unreliable when deployed in challenging time-varying environments. When these systems are used for autonomous decision-making, such as in self-driving vehicles, the impact of their mistakes can be catastrophic. As such, it is important to characterize the performance of the system and predict when and where it may fail in order to take appropriate action. While similar in spirit to the idea of introspection, this work introduces a new paradigm for predicting the likely performance of a robot’s perception system based on past experience in the same workspace. In particular, we propose two models that probabilistically predict perception performance from observations gathered over time. While both approaches are place-specific, the second approach additionally considers appearance similarity when incorporating past observations. We evaluate our method in a classical decision-making scenario in which the robot must choose when and where to drive autonomously in 60km of driving data from an urban environment. Results demonstrate that both approaches lead to fewer false decisions (in terms of incorrectly offering or denying autonomy) for two different detector models, and show that leveraging visual appearance within a state-of-the-art navigation framework increases the accuracy of our performance predictions.Keywords
This publication has 33 references indexed in Scilit:
- Fast Feature Pyramids for Object DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Experience-based navigation for long-term localisationThe International Journal of Robotics Research, 2013
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- Lighting‐invariant Visual Teach and Repeat Using Appearance‐based LidarJournal of Field Robotics, 2012
- Diagnosing Error in Object Detectors2012
- Undoing the Damage of Dataset Bias2012
- Appearance-only SLAM at large scale with FAB-MAP 2.0The International Journal of Robotics Research, 2010
- The Marulan Data Sets: Multi-sensor Perception in a Natural Environment with Challenging ConditionsThe International Journal of Robotics Research, 2010
- Visual teach and repeat for long‐range rover autonomyJournal of Field Robotics, 2010
- Gaussian Processes for Object CategorizationInternational Journal of Computer Vision, 2009