Deep multimodal relational reasoning and guided attention for chart question answering
- 1 November 2024
- journal article
- research article
- Published by SPIE-Intl Soc Optical Eng in Journal of Electronic Imaging
- Vol. 33 (06) , 063052-18
Abstract
The proposed work for chart question answering (CQA) addresses crucial aspects, particularly those associated with questions requiring complex reasoning and visual references to charts and those related to optical character recognition (OCR) noise. Existing methods focused mainly on image features, neglecting the unique characteristics of the visual representation, which consequently not able to answer complex reasoning questions. Introducing structural embedding, question embedding, and a transformer-based relational understanding of the visualizations enable the proposed model to answer complex reasoning questions. The incorporation of the reading component and visual component using BERT encoding leverages both textual and visual information, which overcomes the OCR noise-related issue. The comprehensive ablation study systematically evaluates the different components of our model. Moreover, experiments are conducted on five datasets: DVQA, FigureQA, PlotQA, LEAFQA++, and ChartQA, which makes the assessment robust and generalized. The overall accuracy of our model is increased by 1.15% compared with STL-CQA on the test familiar of DVQA, 0.55% compared with PReFIL on FigureQA, 8.03% compared with Plot-QA, 2.28% compared with STL-CQA test-familiar split of LEAFQA++, and 0.40% compared with VisionTaPas on the test-set of ChartQA, which shows the effectiveness and superiority of the proposed model over existing methods on these datasets.Keywords
This publication has 29 references indexed in Scilit:
- Integrating multimodal features by a two-way co-attention mechanism for visual question answeringMultimedia Tools and Applications, 2023
- Enhanced copy-move forgery detection using deep convolutional neural network (DCNN) employing the ResNet-101 transfer learning modelMultimedia Tools and Applications, 2023
- Classification-Regression for Chart Comprehension2022
- A framework for visual question answering with the integration of scene-text using PHOCs and fisher vectorsExpert Systems with Applications, 2022
- Improving visual question answering by combining scene-text informationMultimedia Tools and Applications, 2022
- A survey of methods, datasets and evaluation metrics for visual question answeringImage and Vision Computing, 2021
- An Improved Attention and Hybrid Optimization Technique for Visual Question AnsweringNeural Processing Letters, 2021
- Image captioning improved visual question answeringMultimedia Tools and Applications, 2021
- Visual question answering model based on graph neural network and contextual attentionImage and Vision Computing, 2021
- FigureSeer: Parsing Result-Figures in Research Papers2016