AMD-Proj: Adaptive Memory-Driven Selective Gradient Projection for Continual Learning in Document Understanding
Open Access
- 23 April 2026
- journal article
- research article
- Published by MDPI AG in Technologies
- Vol. 14 (5) , 250
Abstract
Visually rich document understanding (VrDU) models rely on tightly coupled textual, layout, and visual representations. In real-world deployments, these models must continuously adapt to new document domains over time. However, naïve sequential fine-tuning leads to severe catastrophic forgetting due to shared parameters and strong cross-task interference. Existing continual learning approaches either constrain parameter updates, preserve output distributions, or uniformly suppress gradient directions associated with previous tasks. While effective in limited settings, these strategies fail to balance stability and plasticity in large multimodal transformers. We propose AMD-Proj, an adaptive memory-driven selective gradient projection framework for continual learning in document understanding. It models task knowledge using specific gradient subspaces and adaptively modulates incoming gradients based on their alignment with this memory, selectively blocking interfering directions while reinforcing reusable ones. An efficient truncated SVD mechanism with online subspace merging ensures bounded memory usage and scalability to large transformer-based architectures. We evaluate AMD-Proj on four VrDU benchmarks (FUNSD, SROIE, CORD, and BuDDIE) under a task-incremental learning setting using LayoutLMv2 and LayoutLMv3 backbones. Results show that AMD-Proj reduces catastrophic forgetting and improves F1-based stability over EWC, GPM, LwF, OWM, CUBER, TRGP and parameter-efficient fine-tuning methods. Extensive mechanistic analyses, including gradient spectrum decomposition and layer-wise reuse versus block dynamics, provide insight into how selective gradient projection controls optimization geometry during continual adaptation. These findings establish selective gradient projection as a principled and interpretable approach for continual learning in visually rich document understanding.Keywords
Funding Information
- Leyton
This publication has 25 references indexed in Scilit:
- A differentiable and uncertainty-aware mutual information regularizer for bias mitigationNeurocomputing, 2025
- LMGDoc: Light Multigranular GNN for Efficient Document UnderstandingIEEE Access, 2025
- A Comprehensive Survey of Continual Learning: Theory, Method and ApplicationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Doc2Graph: a Task Agnostic Document Understanding Framework based on Graph Neural NetworksarXiv, 2022
- TRGP: Trust Region Gradient Projection for Continual LearningarXiv, 2022
- Continual learning of context-dependent processing in neural networksNature Machine Intelligence, 2019
- Memory Aware Synapses: Learning What (not) to Forget2018
- Learning without ForgettingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Progressive Neural NetworksarXiv, 2016