Integration of Variational Methods and Deep Learning for Image Processing: A Task-Oriented Review
Open Access
- 1 November 2026
- journal article
- review article
- Published by MDPI AG in Mathematics
- Vol. 14 (21) , 3977
Abstract
Variational methods and deep learning represent two distinct paradigms that have each shaped the development of modern image processing. Variational formulations are grounded in carefully designed energy functionals and partial differential equations (PDEs) that encode prior knowledge about image structure; they offer interpretability, theoretical guarantees, and strong performance. Deep learning, by contrast, learns image priors implicitly from data through highly expressive parametric models and has achieved impressive improvements across a wide range of tasks. Over roughly the past decade, a growing body of work has sought to integrate the two paradigms, combining the mathematical rigor of variational models with the representational power of deep networks. This article provides a task-oriented review of this integration. We survey the principal mathematical frameworks—total variation (TV) and its generalisations, the Mumford–Shah functional, Euler’s elastica, level set methods, and nonlinear diffusion PDEs—and describe how each has been coupled with deep architectures. We then organise the literature by tasks, covering image denoising, segmentation, reconstruction, super-resolution, inpainting, and deblurring. For each task, we survey major learning paradigms and representative deep learning integration frameworks—centred on variational loss formulation, algorithm unrolling, PDE-driven architectures, plug-and-play priors, deep image prior, and score-based generative modelling—highlighting the synergy between mathematically rigorous variational principles and empirical model performance. We further establish theoretical links: residual networks to ordinary differential equations, and score-based diffusion models to stochastic PDEs. Standard benchmarks and metrics are summarised, followed by open challenges such as cross-domain generalisation, robustness, interpretability, hallucination control, and data-efficient learning. Finally, conclusions are drawn based on the available evidence, and the maturity of each research direction is explicitly discussed. This review differs from existing surveys in three respects. It classifies methods by the locus at which the variational and the learned component are coupled—the training objective, the network architecture, or the energy minimised at inference—rather than by task or by assumption on the forward operator. It applies that single classification uniformly across six tasks, including segmentation and inpainting, which carry no explicit forward operator and therefore fall outside the scope of existing surveys of learned inverse problems. Moreover, it uses the resulting coverage map, audited entry by entry against a formal membership criterion, to distinguish combinations that have been realised from those that remain unexplored, those the taxonomy excludes, and those connected only loosely.Keywords
Funding Information
- Open Research Fund of the State Key Laboratory of Brain-Machine Intelligence, Zhejiang University (BMI2500010)
- Key Scientific Research Projects of Colleges and Universities in Anhui Province (2025AHGXZK30179)
- Key Scientific Research Projects of Colleges and Universities in Anhui Province (2025AHGXZK20090)
- Excellent Research Innovation Team of Anhui Provincial Education Department (2023AH010066)
- Peak Disciplines at Bengbu University (2025GFXK01)
- High-Level Talents Research Start-up Funding of Bengbu University (2025GQD021)
- Shenzhen Nature Science Research Foundation Key project (ZDYJ20251211121303005)
- 2024 Guangdong Provincial Educational Science Planning Project (2024GXJK905)
This publication has 124 references indexed in Scilit:
- Simple Baselines for Image Restoration2022
- LevelSet R-CNN: A Deep Variational Method for Instance Segmentation2020
- Self‐supervised learning of physics‐guided reconstruction neural networks without fully sampled reference dataMagnetic Resonance in Medicine, 2020
- Uncertainty-Aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation2019
- Image Inpainting for Irregular Holes Using Partial Convolutions2018
- Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation2018
- Learning a variational network for reconstruction of accelerated MRI dataMagnetic Resonance in Medicine, 2017
- Low‐dose CT for the detection and classification of metastatic liver lesions: Results of the 2016 Low Dose CT Grand ChallengeMedical Physics, 2017
- Mathematical Problems in Image Processing2006
- Optimal approximations by piecewise smooth functions and associated variational problemsCommunications on Pure and Applied Mathematics, 1989