Machine Vision Models for Food Recognition and Portion Estimation

Abstract

Accurate monitoring of dietary intake is essential for public health and nutritional research. Traditional approaches are time-consuming and erroneous. An alternative strategy is provided by automated food recognition and portion estimation systems, but their dependability is yet unclear. This study investigates the potential of multimodal large language models for automating food item identification and nutritional estimation from meal images in a real cafeteria environment, with a focus on how structured contextual information affects model performance. The study employs an experimental comparative methodology using 192 meal images from a university cafeteria research environment in Finland (Flavoria). Two LLM families, which are GPT and Gemini, were evaluated across four analyses covering three contextual conditions: no context, a daily restaurant menu as context(Sodexo), and the Finnish national food composition database as context(Fineli). The performance was evaluated using mean absolute error, mean absolute percentage error, and error distribution analysis against measured ground truth nutritional data. Key findings of this study reveal that both models produced significant estimation errors, without any real contextual grounding. The menu-based context seemed to improve performance quite a bit for both models, but the Fineli database context came across as more helpful for Gemini than for GPT, for carbohydrate estimation. Visually ambiguous food items, such as sauces, were consistently harder to estimate than structured dishes. Food name inconsistency is also a practical issue needing some post-processing normalisation. This study contributes empirical evidence on the feasibility and limitations of multi-modal LLMs for food recognition in food service settings. The results suggest that while contextual grounding meaningfully improves model outputs, current general- purpose LLMs are not yet sufficiently accurate for precise nutritional monitoring. Future work directions include retrieval-augmented generation combining menu and database context, and evaluation on larger and more diverse datasets.