Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
Authors:
- Naitik KhandelwalNKNaitik Khandelwal
- Xiao LiuXLXiao Liu
- Mengmi ZhangMZMengmi Zhang
Preprint
Abstract
Scene graph generation (SGG) involves analyzing images to extract meaningful information about objects and their relationships. Given the dynamic nature of the visual world, it becomes crucial for AI systems to detect new objects and establish their new relationships with existing objects. To address the lack of continual learning methodologies in SGG, we introduce the comprehensive Continual ScenE Graph Generation (CSEGG) dataset along with 3 learning scenarios and 8 evaluation metrics. Our research investigates the continual learning performances of existing SGG methods on the retention of previous object entities and relationships as they learn new ones. Moreover, we also explore how continual object detection enhances generalization in classifying known relationships on unknown objects. We conduct extensive experiments benchmarking and analyzing the classical two-stage SGG methods and the most recent transformer-based SGG methods in continual learning settings, and gain valuable insights into the CSEGG problem. We invite the research community to explore this emerging field of study.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Learning to Learn: How to Continuously Teach Humans and Machines2023
- Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA TaskProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Image generation models from scene graphs and layouts: A comparative analysisJournal of King Saud University Computer and Information Sciences, 2023
- Online continual learning in image classification: An empirical surveyNeurocomputing, 2022
- Uncovering Main Causalities for Long-tailed Information Extraction2021
- The Open Images Dataset V4International Journal of Computer Vision, 2020
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Image retrieval using scene graphs2015
- Recognition using visual phrases2011