A Three-Stage Transformer-Based Approach for Food Mass Estimation
Open Access
- 7 November 2025
- conference paper
- Published by MDPI AG in Engineering Proceedings
- Vol. 118 (1) , 36
Abstract
Accurate food mass estimation is a key component of automated calorie estimation tools, and there is growing interest in leveraging image analysis for this purpose due to its ease of use and scalability. However, current methods face important limitations. Some rely on 3D sensors for depth estimation, which are not widely accessible to all users, while others depend on camera intrinsic parameters to estimate volume, reducing their adaptability across different devices. Furthermore, AI-based approaches that bypass these parameters often struggle with generalizability when applied to images captured using diverse sensors or camera settings. To overcome these challenges, we introduce a three-stage, transformer-based method for estimating food mass from RGB images, balancing accuracy, computational efficiency, and scalability. The first stage applies the Segment Anything Model (SAM 2) to segment food items in images from the SUECFood dataset. Next, we use the Global-Local Path Network (GLPN) to perform monocular depth estimation (MDE) on the Nutrition5k dataset, inferring depth information from a single image. These outputs are then combined through alpha compositing to generate enhanced composite images with precise object boundaries. Finally, a Vision Transformer (ViT) model processes the composite images to estimate food mass by extracting relevant visual and spatial features. Our method achieves notable improvements in accuracy compared to previous approaches, with a mean squared error (MSE) of 5.61 and a mean absolute error (MAE) of 1.07. Notably, this pipeline does not require specialized hardware like depth sensors or multi-view imaging, making it well-suited for practical deployment. Future work will explore the integration of ingredient recognition to support a more comprehensive dietary assessment system.Keywords
This publication has 24 references indexed in Scilit:
- Calorie-Counting Apps for Monitoring and Managing Calorie Intake in Adults Living With Weight-Related Chronic Diseases: Decade-Long Scoping Review (2013-2024)JMIR mHealth and uHealth, 2026
- Image-based food monitoring and dietary management for patients living with diabetes: a scoping review of calorie counting applicationsFrontiers in Nutrition, 2025
- A Review of Image-Based Food Recognition and Volume Estimation Artificial Intelligence SystemsIEEE Reviews in Biomedical Engineering, 2023
- Rapid Non-Destructive Analysis of Food Nutrient Content Using Swin-NutritionFoods, 2022
- Human-Mimetic Estimation of Food Volume from a Single-View RGB Image Using an AI SystemElectronics, 2021
- Point2Volume: A Vision-Based Dietary Assessment Approach Using View SynthesisIEEE Transactions on Industrial Informatics, 2019
- Image-based food portion size estimation using a smartphone without a fiducial markerPublic Health Nutrition, 2018
- A New Approach to Image-Based Estimation of Food VolumeAlgorithms, 2017
- Two-View 3D Reconstruction for Food Volume EstimationIEEE Transactions on Multimedia, 2016
- Measuring Calorie and Nutrition From Food ImageIEEE Transactions on Instrumentation and Measurement, 2014