Improving dietary data quality in nutrition studies: development and validation of an explainable, reproducible, open machine learning-assisted framework for outlier detection in 24-hour recalls
Open Access
- 24 August 2026
- journal article
- Published by Oxford University Press (OUP) in American Journal of Epidemiology
Abstract
Twenty-four-hour dietary recalls (24HRs) are widely used in nutritional epidemiology, but repeated administration is required to capture within-person variability, introducing temporal dependence and challenges for data cleaning. Existing outlier detection approaches are largely static, manual, and based on adult-centered thresholds, limiting reproducibility and scalability. We developed and validated an explainable, reproducible machine learning (ML)-assisted framework to detect and evaluate implausible intake values in cross-sectional and longitudinal 24HRs and to generate artificial intelligence (AI)-ready dietary datasets. The open pipeline integrates automated statistical detection using digitized National Cancer Institute 5th-95th percentile intake ranges, longitudinal IQR-based flagging, and ML-based rule explainability for outlier detection. Validation was conducted using the MIcrobiota, GROWth and Diet study in Ontario, Canada (126 children aged 8-12 years), and data from the 2015 Canadian Community Health Survey-Nutrition, a nationally representative sample of Canadian children and adults with repeated 24HRs (n = 15 216). Across energy intake and 7 nutrients, 75 recalls containing at least 1 potentially implausible value were identified. In longitudinal data, the framework achieved >77% sensitivity, 97% specificity, and 88% precision, outperforming the static method. This explainable data-driven framework improves data quality, transparency, and reproducibility, supporting scalable epidemiologic analyses using AI-ready dietary data.Keywords
Funding Information
- Canadian Institutes of Health Research (PJ9-190386)
- Canadian Institutes of Health Research (PJT-195842)
This publication has 25 references indexed in Scilit:
- Food Diary, Food Frequency Questionnaire, and 24-Hour Dietary Recall2022
- On the nature and types of anomalies: a review of deviations in dataInternational Journal of Data Science and Analytics, 2021
- Dietary assessment toolkits: an overviewPublic Health Nutrition, 2018
- Identifying small groups of foods that can predict achievement of key dietary recommendations: data mining of the UK National Diet and Nutrition Survey, 2008–12Public Health Nutrition, 2016
- The Automated Self-Administered 24-Hour Dietary Recall (ASA24): A Resource for Researchers, Clinicians, and Educators from the National Cancer InstituteJournal of the Academy of Nutrition and Dietetics, 2012
- Number of 24-Hour Diet Recalls Needed to Estimate Energy IntakeAnnals of Epidemiology, 2009
- An adjusted boxplot for skewed distributionsComputational Statistics & Data Analysis, 2008
- A New Statistical Method for Estimating the Usual Intake of Episodically Consumed Foods with Application to Their DistributionJournal of the American Dietetic Association, 2006
- Barnett, V., and Lewis T.: Outliers in Statistical Data. 3rd edition. J. Wiley & Sons 1994, XVII. 582 pp., £49.95Biometrical Journal, 1995
- Differences in precision of dietary estimates among different population subgroupsAnnals of Epidemiology, 1993