Effective Pre-Training Method and Its Compositional Intelligence for Image Captioning
Open Access
Abstract
With the increase in the performance of deep learning models, the model parameter has increased exponentially. An increase in model parameters leads to an increase in computation and training time, i.e., an increase in training cost. To reduce the training cost, we propose Compositional Intelligence (CI). This is a reuse method that combines pre-trained models for different tasks. Since the CI uses a well-trained model, good performance and small training cost can be expected in the target task. We applied the CI to the Image Captioning task. Compared to using a trained feature extractor, the caption generator is usually trained from scratch. On the other hand, we pre-trained the Transformer model as a caption generator and applied CI, i.e., we used a pre-trained feature extractor and a pre-trained caption generator. To compare the training cost of the From Scratch model and the CI model, early stopping was applied during fine-tuning of the image captioning task. On the MS-COCO dataset, the vanilla image captioning model reduced training cost by 13.8% and improved performance by up to 3.2%, and the Object Relation Transformer model reduced training cost by 21.3%.Keywords
Funding Information
- Korea Government (MSIT) (No. 2020-0-01373, No. 2018R1A5A7059549, No. 2020R1A2C1014037)
- National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2020-0-01373, No. 2018R1A5A7059549, No. 2020R1A2C1014037)
- National Research Foundation of Korea (NRF) (No. 2020-0-01373, No. 2018R1A5A7059549, No. 2020R1A2C1014037)
This publication has 34 references indexed in Scilit:
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Show and tell: A neural image caption generator2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- NLTK2002
- Long Short-Term MemoryNeural Computation, 1997