Experts-Guided Unbalanced Optimal Transport for ISP Learning from Unpaired and/or Paired Data
Authors:
Preprint
Abstract
Learned Image Signal Processing (ISP) pipelines offer powerful end-to-end performance but are critically dependent on large-scale paired raw-to-sRGB datasets. This reliance on costly-to-acquire paired data remains a significant bottleneck. To address this challenge, we introduce a novel, unsupervised training framework based on Optimal Transport capable of training arbitrary ISP architectures in both unpaired and paired modes. We are the first to successfully apply Unbalanced Optimal Transport (UOT) for this complex, cross-domain translation task. Our UOT-based framework provides robustness to outliers in the target sRGB data, allowing it to discount atypical samples that would be prohibitively costly to map. A key component of our framework is a novel ``committee of expert discriminators,'' a hybrid adversarial regularizer. This committee guides the optimal transport mapping by providing specialized, targeted gradients to correct specific ISP failure modes, including color fidelity, structural artifacts, and frequency-domain realism. To demonstrate the superiority of our approach, we retrained existing state-of-the-art ISP architectures using our paired and unpaired setups. Our experiments show that while our framework, when trained in paired mode, exceeds the performance of the original paired methods across all metrics, our unpaired mode concurrently achieves quantitative and qualitative performance that rivals, and in some cases surpasses, the original paired-trained counterparts. The code and pre-trained models are available at: https://github.com/gosha20777/EGUOT-ISP.git.Keywords
All Related Versions
This publication has 42 references indexed in Scilit:
- An Optimal Transport Perspective on Unpaired Image Super-ResolutionJournal of Optimization Theory and Applications, 2025
- Rawformer: Unpaired Raw-to-Raw Translation for Learnable Camera ISPs2024
- MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning2023
- RAWtoBit: A Fully End-to-end Camera ISP Network2022
- SepLUT: Separable Image-Adaptive Lookup Tables for Real-Time Image Enhancement2022
- Fast and High Quality Image Denoising via Malleable Convolution2022
- Unpaired Image-to-Image Translation Using Adversarial Consistency Loss2020
- Contrastive Learning for Unpaired Image-to-Image Translation2020
- AWNet: Attentive Wavelet Network for Image ISP2020
- AIM 2020 Challenge on Learned Image Signal Processing Pipeline2020