Unbiased Scene Graph Generation via Two-Stage Causal Modeling
- 12 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 45 (10) , 12562-12580
Abstract
Despite the impressive performance of recent unbiased Scene Graph Generation (SGG) methods, the current debiasing literature mainly focuses on the long-tailed distribution problem, whereas it overlooks another source of bias, i.e. , semantic confusion, which makes the SGG model prone to yield false predictions for similar relationships. In this paper, we explore a debiasing procedure for the SGG task leveraging causal inference. Our central insight is that the Sparse Mechanism Shift (SMS) in causality allows independent intervention on multiple biases, thereby potentially preserving head category performance while pursuing the prediction of high-informative tail relationships. However, the noisy datasets lead to unobserved confounders for the SGG task, and thus the constructed causal models are always causal-insufficient to benefit from SMS. To remedy this, we propose Two-stage Causal Modeling (TsCM) for the SGG task, which takes the long-tailed distribution and semantic confusion as confounders to the Structural Causal Model (SCM) and then decouples the causal intervention into two stages. The first stage is causal representation learning, where we use a novel Population Loss (P-Loss) to intervene in the semantic confusion confounder. The second stage introduces the Adaptive Logit Adjustment (AL-Adjustment) to eliminate the long-tailed distribution confounder to complete causal calibration learning. These two stages are model agnostic and thus can be used in any SGG model that seeks unbiased predictions. Comprehensive experiments conducted on the popular SGG backbones and benchmarks show that our TsCM can achieve state-of-the-art performance in terms of mean recall rate. Furthermore, TsCM can maintain a higher recall rate than other debiasing methods, which indicates that our method can achieve a better tradeoff between head and tail relationships.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2021YFB3100800)
- Research Council of Finland (331883)
- Infotech Project FRAGES
- National Natural Science Foundation of China (62376283, 61872379, 62022091, 62201588, 62201603)
This publication has 48 references indexed in Scilit:
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studiesStatistics in Medicine, 2015
- Image retrieval using scene graphs2015
- Learning semantic relationships for better action retrieval in images2015
- Review of inverse probability weighting for dealing with missing dataStatistical Methods in Medical Research, 2011
- A note on margin-based loss functions in classificationStatistics & Probability Letters, 2004
- Statistics and causal inference: A reviewTEST, 2003
- Causal diagrams for empirical researchBiometrika, 1995
- Contour integration by the human visual system: Evidence for a local “association field”Vision Research, 1993
- The Logit Model and Response-Based SamplesSociological Methods & Research, 1989