Active Learning Inspired ControlNet Guidance for Augmenting Semantic Segmentation Datasets
Preprint
Abstract
Recent advances in conditional image generation from diffusion models have shown great potential in achieving impressive image quality while preserving the constraints introduced by the user. In particular, ControlNet enables precise alignment between ground truth segmentation masks and the generated image content, allowing the enhancement of training datasets in segmentation tasks. This raises a key question: Can ControlNet additionally be guided to generate the most informative synthetic samples for a specific task? Inspired by active learning, where the most informative real-world samples are selected based on sample difficulty or model uncertainty, we propose the first approach to integrate active learning-based selection metrics into the backward diffusion process for sample generation. Specifically, we explore uncertainty, query by committee, and expected model change, which are commonly used in active learning, and demonstrate their application for guiding the sample generation process through gradient approximation. Our method is training-free, modifying only the backward diffusion process, allowing it to be used on any pretrained ControlNet. Using this process, we show that segmentation models trained with guided synthetic data outperform those trained on non-guided synthetic data. Our work underscores the need for advanced control mechanisms for diffusion-based models, which are not only aligned with image content but additionally downstream task performance, highlighting the true potential of synthetic data generation.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Dataset Enhancement with Instance-Level Augmentations2024
- Diffusion Models for Imperceptible and Transferable Adversarial AttackIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Ten Years of Active Learning Techniques and Object Detection: A Systematic ReviewApplied Sciences, 2023
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion ModelsACM Transactions on Graphics, 2023
- Content-based Unrestricted Adversarial Attack2023
- Dataset Diffusion: Diffusion-based Synthetic Data Generation for Pixel-Level Semantic Segmentation2023
- Sequential Operations in Digital Picture ProcessingJournal of the ACM, 1966