Ho-Young Jung
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Ho-Young Jung has a total of 28 co-authors. Here are the top co-authors by number of shared publications:
- June-Woo Kim
GIST InnoCORE AI-Nano Convergence Institute for Early Detection of Neurodegenerative Disease, Gwangju Institute of Science and Technology, Gwangju, Korea
6 shared publications - Jeong Woo Seo
ICT-Connected Composite Intelligence Research Center, Kyungpook National University, Daegu 41566, Republic of Korea
3 shared publications June-Woo Kim
Department of Artificial Intelligence, Kyungpook National University, Daegu, South Korea
3 shared publications- Hoon Chung
Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea
2 shared publications - Hyekyung Yoon
Department of Artificial Intelligence, Graduate School, Kyungpook National University, Daegu 41566, Korea
2 shared publications Suyoung Kim
2 shared publications
Latest Publications
- Open Access
- 2 November 2026
- Published by MDPI AG in Applied Sciences
- Vol. 16 (21) , 10836
KoDocXAI: A Quantitative Evaluation Framework and Benchmark for Explainability in Key Information Extraction from Korean Administrative Form Documents
Key information extraction (KIE) models are increasingly used in high-stakes document workflows, where each extracted value must be accompanied by evidence a reviewer can verify. Evaluating their explanations is difficult: human rationale annotation is costly, and in agglutinative Korean, subword tokens misalign with the units humans read. We present KoDocXAI, an evaluation framework and pilot benchmark for the explanations of KIE models on Korean administrative form documents. It derives a value-region rationale reference from existing KIE bounding-box annotations and scores value-region plausibility, whether or not an explanation localizes the extracted value; re-aggregates subword attributions to morphemes and eojeols (Korean spacing units); and measures faithfulness by modality-wise perturbation with the pretraining [MASK] token while protecting the ground-truth span. The pilot benchmark v1.0 covers five fields over 300 permit-and-authorization dispatch documents, 20% of which are doubly annotated. With a fine-tuned LayoutXLM, Integrated Gradients exceeds a random baseline on both axes, whereas attention rollout stays at baseline level under both a first-token and a span-level target. Span protection removes inflated faithfulness scores and a spurious aggregation-level effect. The modality interventions differed in strength, but under them the image channel contributed practically nothing, confining the validated scope to text and layout. These results hold for this document type and model; they support validating the measurement protocol itself and reporting explainability separately by axis and modality.
- 2 November 2026
- Published by MDPI AG in Applied Sciences
- Vol. 16 (21) , 10836
Open AccessKoDocXAI: A Quantitative Evaluation Framework and Benchmark for Explainability in Key Information Extraction from Korean Administrative Form Documents
Key information extraction (KIE) models are increasingly used in high-stakes document workflows, where each extracted value must be accompanied by evidence a reviewer can verify. Evaluating their explanations is difficult: human rationale annotation is costly, and in agglutinative Korean, subword tokens misalign with the units humans read. We present KoDocXAI, an evaluation framework and pilot benchmark for the explanations of KIE models on Korean administrative form documents. It derives a value-region rationale reference from existing KIE bounding-box annotations and scores value-region plausibility, whether or not an explanation localizes the extracted value; re-aggregates subword attributions to morphemes and eojeols (Korean spacing units); and measures faithfulness by modality-wise perturbation with the pretraining [MASK] token while protecting the ground-truth span. The pilot benchmark v1.0 covers five fields over 300 permit-and-authorization dispatch documents, 20% of which are doubly annotated. With a fine-tuned LayoutXLM, Integrated Gradients exceeds a random baseline on both axes, whereas attention rollout stays at baseline level under both a first-token and a span-level target. Span protection removes inflated faithfulness scores and a spurious aggregation-level effect. The modality interventions differed in strength, but under them the image channel contributed practically nothing, confining the validated scope to text and layout. These results hold for this document type and model; they support validating the measurement protocol itself and reporting explainability separately by axis and modality.
- Open Access
- 22 October 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (21) , 3852
FlashAttention-Compatible Modality-Mutual Attention with Virtual Reordering
Multimodal large language models (MLLMs) process an interleaved image–text sequence with a single autoregressive backbone under causal attention. Since the causal mask prevents image tokens from attending to the question that follows them, their representations are computed without access to the query, which has been identified as a source of visual hallucination. Modality-mutual attention (MMA) mitigates this issue by unlocking the image-to-question region of the causal mask, and dual-order training (DOT) mitigates it by additionally training on the reversed text-first order. However, MMA turns the global causal mask into a sample-dependent dense mask. Consequently, attention cannot be executed via FlashAttention, which is the most efficient attention path, and falls back to a slower implementation. Furthermore, the positional information in the unlocked region does not match that of the reversed input order that motivates the unlock, that is, the positional semantics are inconsistent. To address both problems, we propose FlashAttention-compatible MMA with virtual reordering (FMMA-VR), which repairs the positional information of the unlocked region as if the two modalities were reordered, without physically reordering the sequence. The proposed method is implemented as a Triton-based region-aware FlashAttention-style kernel. On identically pretrained models, it accelerates the attention path by up to over conventional MMA for long sequences during supervised fine-tuning (SFT). In post-SFT evaluations, it stays on par with conventional MMA across ten benchmark metrics under full fine-tuning. Under low-budget LoRA adaptation, where the positional shift of conventional MMA is not absorbed, FMMA-VR additionally improves the hallucination-sensitive metrics by large margins.- 22 October 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (21) , 3852
Open AccessFlashAttention-Compatible Modality-Mutual Attention with Virtual Reordering
Multimodal large language models (MLLMs) process an interleaved image–text sequence with a single autoregressive backbone under causal attention. Since the causal mask prevents image tokens from attending to the question that follows them, their representations are computed without access to the query, which has been identified as a source of visual hallucination. Modality-mutual attention (MMA) mitigates this issue by unlocking the image-to-question region of the causal mask, and dual-order training (DOT) mitigates it by additionally training on the reversed text-first order. However, MMA turns the global causal mask into a sample-dependent dense mask. Consequently, attention cannot be executed via FlashAttention, which is the most efficient attention path, and falls back to a slower implementation. Furthermore, the positional information in the unlocked region does not match that of the reversed input order that motivates the unlock, that is, the positional semantics are inconsistent. To address both problems, we propose FlashAttention-compatible MMA with virtual reordering (FMMA-VR), which repairs the positional information of the unlocked region as if the two modalities were reordered, without physically reordering the sequence. The proposed method is implemented as a Triton-based region-aware FlashAttention-style kernel. On identically pretrained models, it accelerates the attention path by up to over conventional MMA for long sequences during supervised fine-tuning (SFT). In post-SFT evaluations, it stays on par with conventional MMA across ten benchmark metrics under full fine-tuning. Under low-budget LoRA adaptation, where the positional shift of conventional MMA is not absorbed, FMMA-VR additionally improves the hallucination-sensitive metrics by large margins. - Open Access
- 22 August 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (17) , 3026
Quality-Aware Selection for Retrieval-Augmented Fine-Tuning of Small Language Models
Retrieval-augmented fine-tuning (RAFT) can improve small language models (sLMs) on retrieval-grounded question answering, but the synthetic training data produced by commercial large language models (LLMs) vary in quality. This paper contributes a quality-aware selection protocol—rather than a new RAFT or QLoRA method—that scores LLM-generated alternatives along four embedding-based dimensions (question relevance, answer faithfulness, QA coherence, and semantic similarity) and selects one alternative per task before parameter-efficient fine-tuning. Under pre-specified paired-bootstrap contrasts with Holm correction, the parameter-free faithfulness-based selector only-AF significantly exceeds random selection on Gemma-2-9B-IT (ΔF1 = +0.106, 95% CI [+0.043, +0.174], Holm-corrected p = 0.019), and its pre-specified weighted companion af-70 (wAF = 0.70) shows the same confirmed pattern (Holm-corrected p = 0.002). Both effects persist under a Korean character-level F1 that removes particles and punctuation (Holm-corrected p = 0.004 and p = 0.042), indicating robustness to the choice of lexical metric. Relative to training on the full 150-row augmented pool, the quality-selected 50-row sets are statistically indistinguishable while using one third of the training data, which we interpret as data efficiency rather than superiority. Across six instruction-tuned models (2B–27B), a significant selector-by-model interaction indicates that the optimal quality axis is model-dependent, and the two smallest models show no benefit from selection. The study’s confirmatory contrasts use a small controlled Korean corpus under a transductive design; two pre-registered validation experiments probe external validity. On an independent five-fold larger corpus with a passage-level train/test split, fine-tuning transfers strongly and the selected one-third subsets show no significant difference from the full pool, while the advantage over random selection is directionally positive but small and not significant; under controlled corruption of 35% of the pool, the metrics detect the damaged rows, and for the score-sum selector the selection-versus-random benefit is significantly larger than on the clean pool (difference-in-differences p = 0.0014; directionally consistent but not significant for the faithfulness selectors). Within this scope, quality-aware selection is a promising, data-efficient safeguard for synthetic RAFT data—performing comparably to full-pool training at one third of the cost, with growing value as pool quality degrades—and larger-scale external validation remains future work.
- 22 August 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (17) , 3026
Open AccessQuality-Aware Selection for Retrieval-Augmented Fine-Tuning of Small Language Models
Retrieval-augmented fine-tuning (RAFT) can improve small language models (sLMs) on retrieval-grounded question answering, but the synthetic training data produced by commercial large language models (LLMs) vary in quality. This paper contributes a quality-aware selection protocol—rather than a new RAFT or QLoRA method—that scores LLM-generated alternatives along four embedding-based dimensions (question relevance, answer faithfulness, QA coherence, and semantic similarity) and selects one alternative per task before parameter-efficient fine-tuning. Under pre-specified paired-bootstrap contrasts with Holm correction, the parameter-free faithfulness-based selector only-AF significantly exceeds random selection on Gemma-2-9B-IT (ΔF1 = +0.106, 95% CI [+0.043, +0.174], Holm-corrected p = 0.019), and its pre-specified weighted companion af-70 (wAF = 0.70) shows the same confirmed pattern (Holm-corrected p = 0.002). Both effects persist under a Korean character-level F1 that removes particles and punctuation (Holm-corrected p = 0.004 and p = 0.042), indicating robustness to the choice of lexical metric. Relative to training on the full 150-row augmented pool, the quality-selected 50-row sets are statistically indistinguishable while using one third of the training data, which we interpret as data efficiency rather than superiority. Across six instruction-tuned models (2B–27B), a significant selector-by-model interaction indicates that the optimal quality axis is model-dependent, and the two smallest models show no benefit from selection. The study’s confirmatory contrasts use a small controlled Korean corpus under a transductive design; two pre-registered validation experiments probe external validity. On an independent five-fold larger corpus with a passage-level train/test split, fine-tuning transfers strongly and the selected one-third subsets show no significant difference from the full pool, while the advantage over random selection is directionally positive but small and not significant; under controlled corruption of 35% of the pool, the metrics detect the damaged rows, and for the score-sum selector the selection-versus-random benefit is significantly larger than on the clean pool (difference-in-differences p = 0.0014; directionally consistent but not significant for the faithfulness selectors). Within this scope, quality-aware selection is a promising, data-efficient safeguard for synthetic RAFT data—performing comparably to full-pool training at one third of the cost, with growing value as pool quality degrades—and larger-scale external validation remains future work.
- Open Access
- 3 July 2026
- Published by Springer Nature in Scientific Reports
- Vol. 16 (1)
A disambiguation framework for refining and answering ambiguous questions
Ambiguous questions often require clarification before reliable answers can be generated. Existing clarification-based question answering systems typically generate answers directly from clarification interactions, implicitly assuming that the required information has been fully recovered. We argue that clarification should instead be viewed as a process of recovering the latent semantic state necessary for answer generation. To this end, we propose SemRec-SV, a semantic-state-centric framework for ambiguity resolution that explicitly models semantic-state recovery, verification, and refinement. The framework consists of FTGate for ambiguity detection, semantic-state recovery from clarification interactions, StateVerify for assessing the sufficiency of the recovered state, and an adaptive refinement mechanism that performs additional interaction only when the recovered state is deemed insufficient. Experiments on the ClarifyingQA benchmark show that SemRec-SV achieves the highest overall performance among the evaluated methods, outperforming clarification-only baselines and a reproduced CLAM baseline across multiple instruction-tuned language models. Additional analyses demonstrate that semantic-state recovery provides benefits beyond clarification alone, state verification improves answer reliability by identifying incompletely recovered information, and adaptive refinement is particularly effective on insufficient cases. These results suggest that ambiguity resolution benefits from explicitly modeling the recovery, verification, and refinement of semantic information rather than treating clarification as a direct path to answer generation.
- 3 July 2026
- Published by Springer Nature in Scientific Reports
- Vol. 16 (1)
Open AccessA disambiguation framework for refining and answering ambiguous questions
Ambiguous questions often require clarification before reliable answers can be generated. Existing clarification-based question answering systems typically generate answers directly from clarification interactions, implicitly assuming that the required information has been fully recovered. We argue that clarification should instead be viewed as a process of recovering the latent semantic state necessary for answer generation. To this end, we propose SemRec-SV, a semantic-state-centric framework for ambiguity resolution that explicitly models semantic-state recovery, verification, and refinement. The framework consists of FTGate for ambiguity detection, semantic-state recovery from clarification interactions, StateVerify for assessing the sufficiency of the recovered state, and an adaptive refinement mechanism that performs additional interaction only when the recovered state is deemed insufficient. Experiments on the ClarifyingQA benchmark show that SemRec-SV achieves the highest overall performance among the evaluated methods, outperforming clarification-only baselines and a reproduced CLAM baseline across multiple instruction-tuned language models. Additional analyses demonstrate that semantic-state recovery provides benefits beyond clarification alone, state verification improves answer reliability by identifying incompletely recovered information, and adaptive refinement is particularly effective on insufficient cases. These results suggest that ambiguity resolution benefits from explicitly modeling the recovery, verification, and refinement of semantic information rather than treating clarification as a direct path to answer generation.
- Open Access
Adapting a Foundation Monocular Depth Model for Soccer Video: From Synthetic Supervision to Match-Level Reliability
Soccer-video analysis centers on pitch-plane tracking, but camera-view depth cues such as occlusion and goal-area structure are not fully represented on the field plane. Synthetic benchmarks provide dense supervision unavailable for real broadcasts, but whether adaptation yields predictions that are reproducible across matches and operationally feasible remains unclear. We evaluate a Depth Anything V2 model adapted to SoccerNet-Depth with four components: Unaligned MDE accuracy, scale-and-shift aligned diagnostic, match-to-match reliability, and accuracy–cost trade-off. The model achieves an unaligned validation AbsRel of 0.00372. The aligned diagnostic shows that Base DAv2 retained substantial scene-depth structure, whereas SoccerNet adaptation enabled direct compatibility with the normalized target without per-frame ground-truth fitting. Relative to the VKITTI-fine-tuned reference, the adaptation improved all eight metrics in all 21 validation matches, with paired Wilcoxon tests significant after Bonferroni correction. On the challenge split, it reduced AbsRel by 34.1% versus the official baseline. The higher-resolution configuration improved the validation AbsRel by 5.9%, while the default retained a better accuracy–cost balance. At 401.57 ms per frame, the default is suited to post-match analysis, not live or near-real-time use. The study contributes a benchmark-scoped adaptation case study and protocol for foundation MDE on SoccerNet-Depth.
Open AccessAdapting a Foundation Monocular Depth Model for Soccer Video: From Synthetic Supervision to Match-Level Reliability
Soccer-video analysis centers on pitch-plane tracking, but camera-view depth cues such as occlusion and goal-area structure are not fully represented on the field plane. Synthetic benchmarks provide dense supervision unavailable for real broadcasts, but whether adaptation yields predictions that are reproducible across matches and operationally feasible remains unclear. We evaluate a Depth Anything V2 model adapted to SoccerNet-Depth with four components: Unaligned MDE accuracy, scale-and-shift aligned diagnostic, match-to-match reliability, and accuracy–cost trade-off. The model achieves an unaligned validation AbsRel of 0.00372. The aligned diagnostic shows that Base DAv2 retained substantial scene-depth structure, whereas SoccerNet adaptation enabled direct compatibility with the normalized target without per-frame ground-truth fitting. Relative to the VKITTI-fine-tuned reference, the adaptation improved all eight metrics in all 21 validation matches, with paired Wilcoxon tests significant after Bonferroni correction. On the challenge split, it reduced AbsRel by 34.1% versus the official baseline. The higher-resolution configuration improved the validation AbsRel by 5.9%, while the default retained a better accuracy–cost balance. At 401.57 ms per frame, the default is suited to post-match analysis, not live or near-real-time use. The study contributes a benchmark-scoped adaptation case study and protocol for foundation MDE on SoccerNet-Depth.
- Open Access
- 14 May 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (10) , 1669
JODAL: Joint Domain Adversarial Learning for TTS-Augmented Automatic Speech Recognition
Despite the rapid advancements in deep learning techniques, the scarcity of task-specific labeled data remains a significant challenge in automatic speech recognition (ASR). While data augmentation approaches and large-scale pretrained speech foundation models have shown promise in mitigating these limitations, they are often insufficient for improving downstream ASR performance in low-resource practical scenarios. We address these challenges by leveraging off-the-shelf text-to-speech (TTS) systems to generate synthetic speech samples using task-specific text and speech prompts in a resynthesis manner. Although we expect that training with both real and synthetic datasets together can mitigate the data scarcity issue while maintaining ASR performance, we observe that the distributional discrepancy between real and synthetic data introduces a domain inconsistency problem, which negatively impacts ASR performance. To mitigate this, we propose a Joint Domain Adversarial Learning (JODAL) method using real and TTS-generated samples that mitigates domain mismatches through domain adversarial training. Experimental results demonstrate that the proposed JODAL consistently improves ASR robustness across various architectures, including Whisper-tiny and three self-supervised models. Notably, for the WavLM architecture, the enhancement caused by the proposed JODAL successfully surpassed the performance of models trained on 100% real data, highlighting its efficacy as a reasonable substitute for expensive real-world data collection.
- 14 May 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (10) , 1669
Open AccessJODAL: Joint Domain Adversarial Learning for TTS-Augmented Automatic Speech Recognition
Despite the rapid advancements in deep learning techniques, the scarcity of task-specific labeled data remains a significant challenge in automatic speech recognition (ASR). While data augmentation approaches and large-scale pretrained speech foundation models have shown promise in mitigating these limitations, they are often insufficient for improving downstream ASR performance in low-resource practical scenarios. We address these challenges by leveraging off-the-shelf text-to-speech (TTS) systems to generate synthetic speech samples using task-specific text and speech prompts in a resynthesis manner. Although we expect that training with both real and synthetic datasets together can mitigate the data scarcity issue while maintaining ASR performance, we observe that the distributional discrepancy between real and synthetic data introduces a domain inconsistency problem, which negatively impacts ASR performance. To mitigate this, we propose a Joint Domain Adversarial Learning (JODAL) method using real and TTS-generated samples that mitigates domain mismatches through domain adversarial training. Experimental results demonstrate that the proposed JODAL consistently improves ASR robustness across various architectures, including Whisper-tiny and three self-supervised models. Notably, for the WavLM architecture, the enhancement caused by the proposed JODAL successfully surpassed the performance of models trained on 100% real data, highlighting its efficacy as a reasonable substitute for expensive real-world data collection.
- 12 May 2026
- Published by Elsevier in Applied Soft Computing
- Vol. 200
Multi-modality based future predictive representation learning for video anticipation captioning
- 12 May 2026
- Published by Elsevier in Applied Soft Computing
- Vol. 200
Multi-modality based future predictive representation learning for video anticipation captioning
- Open Access
- 15 March 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (6) , 996
A Regularized Backbone-Level Cross-Modal Interaction Framework for Stable Temporal Reasoning in Video-Language Models
Deep learning approaches for egocentric video understanding often lack a principled theoretical treatment of stability, particularly when dealing with the sparse, noisy, and temporally ambiguous observations characteristic of first-person imaging. In this work, we frame egocentric video question answering not merely as a classification task, but as an ill-posed inverse problem aimed at reconstructing latent semantic intent from stochastically perturbed visual signals. To address the instability inherent in standard dual-encoder architectures, we present a framework with a mathematical interpretation that incorporates gated cross-modal interaction within the transformer backbone. Formally, the video-side update analyzed in this work is defined as a learnable convex combination of unimodal feature representations and cross-modal attention residuals; the full implementation applies analogous gated cross-modal updates bidirectionally. From a regularization perspective, the gating mechanism can be interpreted as an adaptive parameter that balances data fidelity against language-conditioned structural constraints during feature reconstruction. We provide the Bounded Update Property (Lemma 1) and an analytical layer-wise sensitivity bound and empirically demonstrate that the proposed framework achieves measurable improvements in both accuracy and stability on the EgoTaskQA and MSR-VTT benchmarks. On EgoTaskQA, our model improves accuracy from 27.0% to 31.7% (+4.7 pp) and reduces the accuracy drop under 50% frame drop from 3.93 pp to 0.94 pp. On MSR-VTT, our model improves accuracy by 13.0 pp over the dual-encoder baseline. Under severe perturbation (50% frame drop) on MSR-VTT, our model retains 97.7% of its clean performance, whereas the baseline exhibits near-zero drop accompanied by majority-class behavior. These results provide empirical evidence that the proposed interaction induces stable behavior under perturbations in an ill-posed multimodal inference setting, mitigating sensitivity to sampling variability while preserving query-relevant temporal structure. Furthermore, an entropy-based analysis indicates that the gating mechanism prevents excessive diffusion of attention, promoting coherent temporal reasoning. Overall, this work offers a mathematically informed perspective on designing interaction mechanisms for stable multimodal systems, with a focus on robust reasoning under temporal ambiguity.
- 15 March 2026
- Published by MDPI AG in Mathematics
- Vol. 14 (6) , 996
Open AccessA Regularized Backbone-Level Cross-Modal Interaction Framework for Stable Temporal Reasoning in Video-Language Models
Deep learning approaches for egocentric video understanding often lack a principled theoretical treatment of stability, particularly when dealing with the sparse, noisy, and temporally ambiguous observations characteristic of first-person imaging. In this work, we frame egocentric video question answering not merely as a classification task, but as an ill-posed inverse problem aimed at reconstructing latent semantic intent from stochastically perturbed visual signals. To address the instability inherent in standard dual-encoder architectures, we present a framework with a mathematical interpretation that incorporates gated cross-modal interaction within the transformer backbone. Formally, the video-side update analyzed in this work is defined as a learnable convex combination of unimodal feature representations and cross-modal attention residuals; the full implementation applies analogous gated cross-modal updates bidirectionally. From a regularization perspective, the gating mechanism can be interpreted as an adaptive parameter that balances data fidelity against language-conditioned structural constraints during feature reconstruction. We provide the Bounded Update Property (Lemma 1) and an analytical layer-wise sensitivity bound and empirically demonstrate that the proposed framework achieves measurable improvements in both accuracy and stability on the EgoTaskQA and MSR-VTT benchmarks. On EgoTaskQA, our model improves accuracy from 27.0% to 31.7% (+4.7 pp) and reduces the accuracy drop under 50% frame drop from 3.93 pp to 0.94 pp. On MSR-VTT, our model improves accuracy by 13.0 pp over the dual-encoder baseline. Under severe perturbation (50% frame drop) on MSR-VTT, our model retains 97.7% of its clean performance, whereas the baseline exhibits near-zero drop accompanied by majority-class behavior. These results provide empirical evidence that the proposed interaction induces stable behavior under perturbations in an ill-posed multimodal inference setting, mitigating sensitivity to sampling variability while preserving query-relevant temporal structure. Furthermore, an entropy-based analysis indicates that the gating mechanism prevents excessive diffusion of attention, promoting coherent temporal reasoning. Overall, this work offers a mathematically informed perspective on designing interaction mechanisms for stable multimodal systems, with a focus on robust reasoning under temporal ambiguity.
- 11 March 2026
- Published by Elsevier in Applied Soft Computing
- Vol. 195
Improving training efficiency for length-imbalanced translation data using optimal packing point
- 11 March 2026
- Published by Elsevier in Applied Soft Computing
- Vol. 195
Improving training efficiency for length-imbalanced translation data using optimal packing point
- Open Access
- 20 October 2025
- Published by MDPI AG in Mathematics
- Vol. 13 (20) , 3344
Eliminating Packing-Aware Masking via LoRA-Based Supervised Fine-Tuning of Large Language Models
Packing approaches enhance training efficiency by filling the padding space in each batch with shorter sequences, thereby reducing the total number of batches per epoch. This approach has proven effective in both pre-training and supervised fine-tuning of large language models (LLMs). However, most packing methods necessitate a packing-aware masking (PAM) mechanism to prevent cross-contamination between different text segments in the multi-head attention (MHA) layers. This masking ensures that the scaled dot-product attention operates only within segment boundaries. Despite its functional utility, PAM introduces significant implementation complexity and computational overhead during training. In this paper, we propose a novel method that eliminates the need for PAM during supervised fine-tuning with packing. Instead of masking, we introduce a learnable tensor derived from Low-Rank Adaptation (LoRA) with the query and value parameters of the attention mechanism. This tensor is trained to attenuate the subspace corresponding to cross-contamination, effectively replacing the function of PAM. Through component-wise decomposition of attention head outputs, we isolate the contamination component and demonstrate that it can be attenuated using the LoRA-derived tensor. Empirical evaluations on 7B-scale LLMs show that our method reduces training time and runtime overhead by completely removing the implementation associated with PAM. This enables more scalable and efficient supervised fine-tuning with packing, without compromising model integrity.
- 20 October 2025
- Published by MDPI AG in Mathematics
- Vol. 13 (20) , 3344
Open AccessEliminating Packing-Aware Masking via LoRA-Based Supervised Fine-Tuning of Large Language Models
Packing approaches enhance training efficiency by filling the padding space in each batch with shorter sequences, thereby reducing the total number of batches per epoch. This approach has proven effective in both pre-training and supervised fine-tuning of large language models (LLMs). However, most packing methods necessitate a packing-aware masking (PAM) mechanism to prevent cross-contamination between different text segments in the multi-head attention (MHA) layers. This masking ensures that the scaled dot-product attention operates only within segment boundaries. Despite its functional utility, PAM introduces significant implementation complexity and computational overhead during training. In this paper, we propose a novel method that eliminates the need for PAM during supervised fine-tuning with packing. Instead of masking, we introduce a learnable tensor derived from Low-Rank Adaptation (LoRA) with the query and value parameters of the attention mechanism. This tensor is trained to attenuate the subspace corresponding to cross-contamination, effectively replacing the function of PAM. Through component-wise decomposition of attention head outputs, we isolate the contamination component and demonstrate that it can be attenuated using the LoRA-derived tensor. Empirical evaluations on 7B-scale LLMs show that our method reduces training time and runtime overhead by completely removing the implementation associated with PAM. This enables more scalable and efficient supervised fine-tuning with packing, without compromising model integrity.