Knowing when we don't know: Introspective classification for mission-critical decision making
Authors:
- 1 May 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 2 (10504729) , 4531-4538
Abstract
Classification precision and recall have been widely adopted by roboticists as canonical metrics to quantify the performance of learning algorithms. This paper advocates that for robotics applications, which often involve mission-critical decision making, good performance according to these standard metrics is desirable but insufficient to appropriately characterise system performance. We introduce and motivate the importance of a classifier's introspective capacity: the ability to mitigate potentially overconfident classifications by an appropriate assessment of how qualified the system is to make a judgement on the current test datum. We provide an intuition as to how this introspective capacity can be achieved and systematically investigate it in a selection of classification frameworks commonly used in robotics: support vector machines, LogitBoost classifiers and Gaussian Process classifiers (GPCs). Our experiments demonstrate that for common robotics tasks a framework such as a GPC exhibits a superior introspective capacity while maintaining commensurate classification performance to more popular, alternative approaches.Keywords
This publication has 19 references indexed in Scilit:
- Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognitionNeural Networks, 2012
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- Finding Rare Classes: Adapting Generative and Discriminative Models in Active LearningLecture Notes in Computer Science, 2011
- Gaussian Processes for Object CategorizationInternational Journal of Computer Vision, 2009
- A generative framework for fast urban labeling using spatial and temporal contextAutonomous Robots, 2009
- Curious George: An attentive semantic robotRobotics and Autonomous Systems, 2008
- Supervised semantic labeling of places using information extracted from sensor dataRobotics and Autonomous Systems, 2007
- Stanley: The robot that won the DARPA Grand ChallengeJournal of Field Robotics, 2006
- Discriminative Learning of Markov Random Fields for Segmentation of 3D Scan Data2005
- Bayesian classification with Gaussian processesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998