Are Food Safety Classifiers Learning Hazards or Memorizing Firms? Entity-Level Leakage in FDA Recall Severity Prediction
Preprint
- 4 March 2026
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Machine learning (ML) models for predicting food recall severity could accelerate regulatory triage, yet no systematic benchmark exists on the U.S.\ Food and Drug Administration (FDA) open-access database. We construct the first comprehensive ML benchmark for FDA food recall severity classification (Class I / II / III) using 28,448 enforcement records spanning 2012--2025. A 1,437-dimensional feature space is engineered from TF-IDF and Sentence-BERT embeddings of recall narratives, structured categorical attributes, and temporal indicators. Five classifiers (Logistic Regression, Random Forest, XGBoost, LightGBM, CatBoost) are trained with Optuna-tuned hyperparameters. Under standard random splitting, XGBoost achieves Macro-F1 = 0.89; however, a multi-layer leakage audit reveals that this figure is inflated by entity-level autocorrelation. When firm-aware group splitting, temporal splitting, or their combination is applied, Macro-F1 drops to approximately 0.57. A firm-mode baseline---assigning each company's historically most frequent severity class---reaches 0.82 under random splitting, demonstrating that 92% of the apparent performance stems from firm-level memorisation. Identity-masking experiments confirm that the leakage is structural rather than attributable to explicit company-name tokens. A \( 2 \times 2 \) factorial decomposition shows that firm overlap and temporal continuity are highly collinear; removing either suffices to expose the true generalisation floor. A hazard-type decomposition reveals that pathogen--severity associations transfer across firms, whereas labelling and GMP violations are highly firm-specific, explaining the disproportionate collapse of Class~III prediction under group splitting. SHAP analysis, feature ablation, and a nine-year continuous-learning simulation provide additional insights into model behaviour and retraining strategies. We recommend that food-safety ML studies adopt group-aware or temporal evaluation protocols, report entity-overlap statistics, and include entity-prior baselines to prevent overstated conclusions.
Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Harnessing the power of artificial intelligence (AI) for insights into FDA recall patterns: a retrospective analysisAI & SOCIETY, 2025
- Unveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future DirectionsWIREs Data Mining and Knowledge Discovery, 2025
- AI-Driven Risk Assessment in Food Safety Using EU RASFF DatabaseFood and Bioprocess Technology, 2025
- Analysis of Outbreak Data Reveals Factors Contributing to Salmonellosis Outbreaks Linked to CantaloupesJournal of Food Protection, 2024
- Tabular data: Deep learning is not all you needInformation Fusion, 2022
- A comparison of neural and non-neural machine learning models for food safety risk prediction with European Union RASFF dataFood Control, 2022
- Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AIInformation Fusion, 2020
- Prediction of food fraud type using data from Rapid Alert System for Food and Feed (RASFF) and Bayesian network modellingFood Control, 2016
- Evaluating the costs of meat and poultry recalls to food firms using stock returnsFood Policy, 2016
- The financial impact of product recall announcements in ChinaInternational Journal of Production Economics, 2013