Toward Chinese Food Understanding: A Cross-Modal Ingredient-Level Benchmark
- 11 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 2863-2874
Abstract
Although there are several food-level benchmarks for food-related learning, the lack of fine-grained ingredient annotation significantly impedes progress in food scene understanding. In this study, we focus on Chinese food understanding which involves fine-grained ingredient detection and cross-modal ingredient retrieval. Specifically, to support studies on Chinese food understanding, we build the first cross-modal ingredientlevel dataset called CMIngre, which contains 8,001 image-text pairs from three different sources, i.e. dishes, recipes, and usergenerated content, covering 429 distinct ingredients and 95,290 bounding boxes. Based on CMIngre, we evaluate the performance of traditional CNN-based detection algorithms and transformerbased pre-trained large models for ingredient detection. We also propose baseline methods for the cross-modal ingredient retrieval task in both the end-to-end and two-stage settings. Extensive experiments on CMIngre demonstrate the effectiveness of our proposed methods on food understandingKeywords
Funding Information
- National Natural Science Foundation of China (62202329, U21B2024)
This publication has 48 references indexed in Scilit:
- Vision and Structured-Language Pretraining for Cross-Modal Food RetrievalComputer Vision and Image Understanding, 2024
- A Mobile Food Recognition System for Dietary Assessment2022
- Graph R-CNN for Scene Graph Generation2018
- Stacked Cross Attention for Image-Text Matching2018
- Learning CNN-based Features for Retrieval of Food Images2017
- Learning Cross-Modal Embeddings for Cooking Recipes and Food Images2017
- Cross-Modal Recipe Retrieval: How to Cook this Dish?2016
- Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation2015
- Microsoft COCO: Common Objects in Context2014
- Food-101 – Mining Discriminative Components with Random Forests2014