Controllable Image Captioning with Feature Refinement and Multilayer Fusion

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Image captioning is the task of automatically generating a description of an image. Traditional image captioning models tend to generate a sentence describing the most conspicuous objects, but fail to describe a desired region or object as human. In order to generate sentences based on a given target, understanding the relationships between particular objects and describing them accurately is central to this task. In detail, information-augmented embedding is used to add prior information to each object, and a new Multi-Relational Weighted Graph Convolutional Network (MR-WGCN) is designed for fusing the information of adjacent objects. Then, a dynamic attention decoder module selectively focuses on particular objects or semantic contents. Finally, the model is optimized by similarity loss. The experiment on MSCOCO Entities demonstrates that IANR obtains, to date, the best published CIDEr performance of 124.52% on the Karpathy test split. Extensive experiments and ablations on both the MSCOCO Entities and the Flickr30k Entities demonstrate the effectiveness of each module. Meanwhile, IANR achieves better accuracy and controllability than the state-of-the-art models under the widely used evaluation metric.
Funding Information
  • National Natural Science Foundation of China (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)
  • Key Research and Development Program of Shaanxi (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)
  • Research and development of manufacturing information system platform supporting product lifecycle management (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)
  • Doctoral Research Fund of Xi’an University of Technology (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)
  • Natural Science Foundation of Shaanxi Province (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)
  • Xi’an Science and Technology Foundation (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)
  • Natural Science Foundation of Shaanxi Province (61771386, 2020SF-359, 2018GY-030, 103-451119003, 2021JQ-487, 2019217814GXRC014CG015-GXYD14.11, 2023-JC-YB-550)