UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All
Authors:
- Yuanhuiyi LyuYLYuanhuiyi Lyu
- Xu ZhengXZXu Zheng
- Jiazhou ZhouJZJiazhou Zhou
- Lin WangLWLin Wang
Preprint
Abstract
We present UniBind, a flexible and efficient approach that learns a unified representation space for seven diverse modalities -- images, text, audio, point cloud, thermal, video, and event data. Existing works, eg., ImageBind, treat the image as the central modality and build an image-centered representation space; however, the space may be sub-optimal as it leads to an unbalanced representation space among all modalities. Moreover, the category names are directly used to extract text embeddings for the downstream tasks, making it hardly possible to represent the semantics of multi-modal data. The 'out-of-the-box' insight of our UniBind is to make the alignment center modality-agnostic and further learn a unified and balanced representation space, empowered by the large language models (LLMs). UniBind is superior in its flexible application to all CLIP-style models and delivers remarkable performance boosts. To make this possible, we 1) construct a knowledge base of text embeddings with the help of LLMs and multi-modal LLMs; 2) adaptively build LLM-augmented class-wise embedding center on top of the knowledge base and encoded visual embeddings; 3) align all the embeddings to the LLM-augmented embedding center via contrastive learning to achieve a unified and balanced representation space. UniBind shows strong zero-shot recognition performance gains over prior arts by an average of 6.36%. Finally, we achieve new state-of-the-art performance, eg., a 6.75% gain on ImageNet, on the multi-modal fine-tuning setting while reducing 90% of the learnable parameters.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Enhancing Multi-modal Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation2023
- CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With TransformersIEEE Transactions on Intelligent Transportation Systems, 2023
- Query Rewriting in Retrieval-Augmented Large Language Models2023
- Multimodal Sensors and ML‐Based Data Fusion for Advanced RobotsAdvanced Intelligent Systems, 2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- KAT: A Knowledge Augmented Transformer for Vision-and-Language2022
- Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification2022
- Converting Static Image Datasets to Spiking Neuromorphic Datasets Using SaccadesFrontiers in Neuroscience, 2015