KoDocXAI: A Quantitative Evaluation Framework and Benchmark for Explainability in Key Information Extraction from Korean Administrative Form Documents
Open Access
- 2 November 2026
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 16 (21) , 10836
Abstract
Key information extraction (KIE) models are increasingly used in high-stakes document workflows, where each extracted value must be accompanied by evidence a reviewer can verify. Evaluating their explanations is difficult: human rationale annotation is costly, and in agglutinative Korean, subword tokens misalign with the units humans read. We present KoDocXAI, an evaluation framework and pilot benchmark for the explanations of KIE models on Korean administrative form documents. It derives a value-region rationale reference from existing KIE bounding-box annotations and scores value-region plausibility, whether or not an explanation localizes the extracted value; re-aggregates subword attributions to morphemes and eojeols (Korean spacing units); and measures faithfulness by modality-wise perturbation with the pretraining [MASK] token while protecting the ground-truth span. The pilot benchmark v1.0 covers five fields over 300 permit-and-authorization dispatch documents, 20% of which are doubly annotated. With a fine-tuned LayoutXLM, Integrated Gradients exceeds a random baseline on both axes, whereas attention rollout stays at baseline level under both a first-token and a span-level target. Span protection removes inflated faithfulness scores and a spurious aggregation-level effect. The modality interventions differed in strength, but under them the image channel contributed practically nothing, confining the validated scope to text and layout. These results hold for this document type and model; they support validating the measurement protocol itself and reporting explainability separately by axis and modality.Keywords
This publication has 13 references indexed in Scilit:
- DocXplain: A Novel Model-Agnostic Explainability Method for Document Image Classification2024
- OCR-Free Document Understanding Transformer2022
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking2022
- XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding2022
- LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding2021
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding2020
- ERASER: A Benchmark to Evaluate Rationalized NLP Models2020
- Quantifying Attention Flow in Transformers2020
- ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction2019
- FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents2019