InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12709-12720
Abstract
We present InstructDiffusion, a unified and generic framework for aligning computer vision tasks with hu-man instructions. Unlike existing approaches that integrate prior knowledge and pre-define the output space (e.g., categories and coordinates) for each vision task, we cast diverse vision tasks into a human-intuitive image-manipulating pro-cess whose output space is a flexible and interactive pixel space. Concretely, the model is built upon the diffusion process and is trained to predict pixels according to user instructions, such as encircling the man's left shoulder in red or applying a blue mask to the left car. InstructDiffusion could handle a variety of vision tasks, including understanding tasks (such as segmentation and keypoint de-tection) and generative tasks (such as editing and enhance-ment) and outperforms prior methods on novel datasets. This represents a solid step towards a generalist modeling interface for vision tasks, advancing artificial general intelligence in the field of computer vision.Keywords
All Related Versions
This publication has 40 references indexed in Scilit:
- Simple Baselines for Image Restoration2022
- A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model2022
- A Benchmark and Baseline for Language-Driven Image Editing2021
- GIQA: Generated Image Quality Assessment2020
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- The Open Images Dataset V4International Journal of Computer Vision, 2020
- Modeling Context in Referring Expressions2016
- Learning Deep Structure-Preserving Image-Text Embeddings2016
- The Role of Context for Object Detection and Semantic Segmentation in the Wild2014
- 2D Human Pose Estimation: New Benchmark and State of the Art Analysis2014