IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark
Preprint
Abstract
Real-world infrared imagery presents unique challenges for vision-language models due to the scarcity of aligned text data and domain-specific characteristics. Although existing methods have advanced the field, their reliance on synthetic infrared images generated through style transfer from visible images, which limits their ability to capture the unique characteristics of the infrared modality. To address this, we propose IRGPT, the first multi-modal large language model for real-world infrared images, built upon a large-scale InfraRed-Text Dataset (IR-TD) comprising over 260K authentic image-text pairs. The proposed IR-TD dataset contains real infrared images paired with meticulously handcrafted texts, where the initial drafts originated from two complementary processes: (1) LLM-generated descriptions of visible images, and (2) rule-based descriptions of annotations. Furthermore, we introduce a bi-cross-modal curriculum transfer learning strategy that systematically transfers knowledge from visible to infrared domains by considering the difficulty scores of both infrared-visible and infrared-text. Evaluated on a benchmark of 9 tasks (e.g., recognition, grounding), IRGPT achieves state-of-the-art performance even compared with larger-scale models.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Alignment-Free RGB-T Salient Object Detection: A Large-Scale Dataset and Progressive Correlation NetworkProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Alignment-Free RGBT Salient Object Detection: Semantics-Guided Asymmetric Correlation Network and a Unified BenchmarkIEEE Transactions on Multimedia, 2024
- Vision-Language Models for Vision Tasks: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Quilt-1M: One Million Image-Text Pairs for HistopathologyAdvances in neural information processing systems, 2023
- The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models2022
- Rethinking Maximum Mean Discrepancy for Visual Domain AdaptationIEEE Transactions on Neural Networks and Learning Systems, 2021
- A Comprehensive Survey on Transfer LearningProceedings of the IEEE, 2021