Spuriosity Rankings for Free: A Simple Framework for Last Layer Retraining Based on Object Detection
Preprint
Abstract
Deep neural networks have exhibited remarkable performance in various domains. However, the reliance of these models on spurious features has raised concerns about their reliability. A promising solution to this problem is last-layer retraining, which involves retraining the linear classifier head on a small subset of data without spurious cues. Nevertheless, selecting this subset requires human supervision, which reduces its scalability. Moreover, spurious cues may still exist in the selected subset. As a solution to this problem, we propose a novel ranking framework that leverages an open vocabulary object detection technique to identify images without spurious cues. More specifically, we use the object detector as a measure to score the presence of the target object in the images. Next, the images are sorted based on this score, and the last-layer of the model is retrained on a subset of the data with the highest scores. Our experiments on the ImageNet-1k dataset demonstrate the effectiveness of this ranking framework in sorting images based on spuriousness and using them for last-layer retraining.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Spurious Features Everywhere - Large-Scale Detection of Harmful Spurious Features in ImageNet2023
- MDETR - Modulated Detection for End-to-End Multi-Modal Understanding2021
- Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects2019
- ImageNet: A large-scale hierarchical image database2009