Common Limitations of Image Processing Metrics: A Picture Story

Authors:
  • ARAnnika Reinke
    Annika Reinke
  • MEMatthias Eisenmann
    Matthias Eisenmann
  • MTMinu D. Tizabi
    Minu D. Tizabi
  • CSCarole H. Sudre
    Carole H. Sudre
  • TRTim Rädsch
    Tim Rädsch
  • MAMichela Antonelli
    Michela Antonelli
  • TATal Arbel
    Tal Arbel
  • SBSpyridon Bakas
    Spyridon Bakas
  • MCM. Jorge Cardoso
    M. Jorge Cardoso
  • VCVeronika Cheplygina
    Veronika Cheplygina
  • KFKeyvan Farahani
    Keyvan Farahani
  • BGBen Glocker
    Ben Glocker

Abstract
While the importance of automatic image analysis is increasing at an enormous pace, recent meta-research revealed major flaws with respect to algorithm validation. Specifically, performance metrics are key for objective, transparent and comparative performance assessment, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. A common mission of several international initiatives is therefore to provide researchers with guidelines and tools to choose the performance metrics in a problem-aware manner. This dynamically updated document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts.