Purrfessor: A Fine-tuned Multimodal LLaVA Diet Health Chatbot (Preprint)

Authors:
Preprint
Publisher website
Google Scholar
Abstract
BACKGROUND The integration of Large Language-and-Vision Assistant (LLaVA) models with food and nutrition data enables multimodal meal analysis and contextual dietary guidance. Despite this potential, the reliability and practical usefulness of such systems for supporting everyday dietary decision-making remain underexplored. OBJECTIVE This study introduces Purrfessor, an innovative AI chatbot designed to provide personalized dietary guidance through interactive, multimodal engagement. The study aimed to evaluate its performance in ingredient recognition and recipe generation. METHODS The Purrfessor chatbot was trained using a combination of the FoodData Central database from the USDA, the Recipe2img dataset featuring food images and corresponding recipes, a curated human-annotated dataset derived from Recipe1M, and a customized Q&A dialogue dataset. The system operates under a session-based, multi-turn interaction paradigm, with memory retained only within an active session and no cross-session memory persistence. We implemented a two-phase evaluation framework combining AI-based performance assessment and human scoring. RESULTS Purrfessor achieved a high average cosine similarity of 0.90 in ingredient recognition with human-coded references. In GPT-4.1–based evaluation of recipe generation quality, Purrfessor outperformed the raw LLaVA model across all evaluated dimensions, with the largest improvements in completeness (7.44 vs 6.52), consistency (8.90 vs 7.81), and clarity (9.13 vs 8.39). Overall recipe quality improved from 7.66 to 8.35. Automatic metrics indicated strong ingredient coverage (0.78) and moderate step complexity (0.74), with lower coherence (0.62) and temperature/time specification (0.59), yielding an overall structured score of 0.68. Human evaluators rated Purrfessor’s Q&A accuracy highly: correctness (M=8.71, SD=1.15), relevance (M=9.99, SD=0.10), and clarity (M=9.33, SD=0.68). Error analysis indicated that 56% of responses contained minor hallucinations (i.e., inclusion of inferred secondary details or invisible garnishes). At the same time, core food identification and overall recipe logic remained accurate. CONCLUSIONS Findings highlight the role of anthropomorphic chatbot design and multimodal AI in supporting engaging dietary health conversations. This study offers an example of AI-driven, evidence-based dietary guidance and underscores the potential of health chatbots to nudge informed health decision-making. Insights contribute to the development of digital health interventions and personalized health communication strategies, with implications for the design of engaging, user-centered AI health assistants. CLINICALTRIAL

This publication has 29 references indexed in Scilit: