About MealSense

Inspiration

Food is one of the most important parts of our daily lives, yet most people have very little understanding of the nutritional value of what they eat. Tracking daily food intake often requires manually searching for food items, estimating portion sizes, identifying ingredients, and entering nutritional information. This process is time-consuming, inconvenient, and difficult to maintain consistently.

The problem becomes more significant for people who need to monitor their diets because of specific health conditions, dietary restrictions, or personal fitness goals. In Bangladesh, where traditional foods such as biryani, khichuri, haleem, hilsa curry, bhorta, and various street foods are common, generic international nutrition-tracking applications often fail to recognize local dishes accurately.

We were inspired by the idea of making nutrition tracking more accessible, intelligent, and relevant to everyday eating habits. Instead of asking users to manually describe everything on their plates, we wanted to build a system that could understand food through images and transform that visual information into meaningful nutritional insights.

This idea led to MealSense: an AI-powered food recognition and personalized nutrition tracking platform.

Our goal is not simply to identify food from an image. We want to bridge the gap between food recognition and practical nutritional understanding by combining computer vision, multimodal AI, nutritional data, and personalized recommendations.

What We Built

MealSense is a web-based AI nutrition assistant that helps users understand what they eat through food image analysis.

The central workflow starts when a user uploads a photograph of a meal. Our food classification model analyzes the image and predicts the corresponding dish. The identified food is then passed to a multimodal vision-language model, which helps interpret the dish, identify its likely ingredients, estimate nutritional information, and generate structured nutrition data.

The platform is designed around several core capabilities:

  • AI Food Recognition: Identifies food items from uploaded images, with particular attention to Bangladeshi and regional cuisine.
  • Ingredient Understanding: Uses multimodal AI to interpret recognized dishes and identify their likely ingredients.
  • Nutrition Estimation: Generates nutritional estimates, including calories, carbohydrates, protein, fats, and other relevant nutritional information.
  • Meal Tracking: Allows users to maintain a record of their meals and monitor their daily food consumption.
  • Personalized Nutrition: Aims to provide food suggestions based on individual dietary preferences and health-related restrictions.
  • Doctor-Sharing Workflow: Includes a privacy-conscious approach where users can decide whether to share their food logs with healthcare professionals.
  • Structured Meal Management: Supports saving meals and their associated food items so users can maintain a consistent nutrition history.

An important design consideration is that food recognition and nutritional estimation are not the same problem. Recognizing a dish does not automatically reveal its exact ingredients, preparation method, or portion size. MealSense therefore separates these tasks into multiple AI processing stages rather than relying on a single image-classification prediction.

How We Built It

We approached MealSense as a combination of traditional machine learning, multimodal AI, and web application engineering.

Rather than depending entirely on a large general-purpose AI model, we designed a multi-stage AI pipeline in which specialized models handle different parts of the problem.

1. Food Classification with Computer Vision

We developed a dedicated food image classification model using a pretrained DINOv3 vision backbone.

The model was adapted to recognize food categories relevant to our target users, including:

  • Biryani
  • Beguni
  • Bakarkhani
  • Chola (Chickpeas)
  • Egg Omelette
  • Fuchka
  • Haleem
  • Hilsa
  • Kabab
  • Kala Bhuna
  • Kacha Golla

We used transfer learning to adapt the pretrained vision model to our food dataset rather than training an entire vision model from scratch.

This approach allowed us to leverage pretrained visual representations while specializing the classifier for regional food recognition.

2. Multimodal AI for Nutrition Understanding

Food classification alone cannot provide sufficient nutritional information. For example, identifying a dish as biryani does not reveal its ingredients, cooking method, or serving size.

To address this, we developed a second AI processing stage using a vision-language model (VLM).

We explored parameter-efficient fine-tuning using LoRA to adapt pretrained multimodal models to our food-to-nutrition mapping task.

The intended pipeline takes the food image and its predicted classification as input and generates structured information about:

  • Likely ingredients and their quantities.
  • Estimated serving size and food weight.
  • Approximate calorie content.
  • Macronutrients such as protein, carbohydrates, and fats.
  • Additional nutritional information.

The model is designed to produce structured JSON outputs that can be processed by the backend and displayed through the application.

We also recognized the importance of using reliable nutritional references instead of treating language-model predictions as authoritative nutritional facts.

3. Full-Stack Web Application

We developed MealSense as a web application using a modern frontend-backend architecture.

Technology stack:

Component Technology
Frontend React
Backend Django
API Architecture REST API
Database PostgreSQL
Computer Vision PyTorch, DINOv3
Multimodal AI Vision-Language Models
Fine-tuning LoRA
Data Format Structured JSON

Django handles application logic, user data, meal records, and communication with the AI models. React provides the user interface for uploading meals, viewing nutritional information, and managing food logs.

We also designed the system so that AI inference can be integrated into the existing backend without coupling the entire application to a single model implementation.

4. Personalized Nutrition and Privacy

We wanted MealSense to go beyond being another calorie counter.

Our approach includes personalized food suggestions that consider users' dietary preferences and health-related restrictions.

We also designed a doctor-sharing workflow in which healthcare professionals can request access to a user's food records. The user retains control and can accept or reject the request.

Privacy was another important consideration. Personal meal photographs are treated separately from general food catalogue images, helping prevent private user uploads from becoming publicly accessible food assets.

What We Learned

Building MealSense gave us practical experience in several areas of AI and software engineering.

1. Combining Computer Vision with Multimodal AI

We learned that different AI models have different strengths. A dedicated image classifier can identify a dish efficiently, while a vision-language model can provide richer contextual understanding. Combining them creates a more specialized pipeline than relying on one general-purpose model.

2. Transfer Learning and Parameter-Efficient Fine-Tuning

Working with pretrained models helped us understand how to adapt existing AI architectures to domain-specific problems. We explored DINOv3 for visual classification and LoRA for adapting multimodal models without requiring expensive full-model training.

3. AI Integration into Real Applications

Training a model is only one part of building an AI product. We learned that preprocessing, model loading, inference management, API integration, validation, and output handling are equally important.

A model that performs well inside a notebook must still be engineered carefully before it can become a reliable application component.

4. Structured AI Outputs

We learned the importance of converting unstructured model responses into validated, predictable data structures. Structured JSON makes it easier to integrate AI predictions with databases, APIs, and frontend components.

5. Responsible Nutrition Estimation

We also learned that nutritional estimation from images involves uncertainty. Portion sizes, hidden ingredients, oil content, and cooking techniques can significantly change nutritional values.

Therefore, AI-generated nutrition information should be treated as an estimate rather than a laboratory-verified measurement.

Challenges We Faced

Developing MealSense involved several technical challenges.

1. Recognizing Regional Food Items

One of our biggest challenges was recognizing Bangladeshi dishes accurately.

Many pretrained food recognition models are trained primarily on internationally popular food categories. They may not adequately represent regional dishes, variations in preparation, or visually similar foods.

We addressed this by developing a specialized classification pipeline using a pretrained vision backbone and a domain-specific food dataset.

2. Connecting Food Recognition with Nutrition Estimation

A classification model only predicts a food category. It does not understand the complete nutritional composition of a meal.

We needed an additional processing stage that could interpret ingredients, estimate portions, and generate nutritional information.

Designing this multi-stage pipeline while maintaining consistency between model outputs was a significant challenge.

3. Portion Size and Nutritional Accuracy

Estimating food weight and nutritional values from a single photograph is inherently difficult.

Two visually similar portions can have different weights, ingredients, and nutritional compositions.

We approached this as an estimation problem and designed the pipeline to provide structured nutritional predictions rather than presenting image-based estimates as exact measurements.

4. Model Deployment and Resource Constraints

Developing AI models in notebooks and integrating them into a Django application introduced another set of challenges.

Model checkpoints, dependencies, preprocessing requirements, inference latency, and memory consumption all needed to be considered.

We had to think beyond model accuracy and consider how the models would operate within an actual web application.

5. Maintaining Data Privacy

Meal photographs and nutrition histories can contain sensitive personal information.

We therefore considered access control, private image storage, and user-controlled doctor sharing as important parts of the application architecture rather than optional additions.

Real-World Impact

We believe MealSense can make nutrition awareness more accessible by reducing the effort required to understand everyday meals.

For individuals, it can simplify food logging and help them become more conscious of their eating habits.

For people with dietary restrictions, it can provide a foundation for more personalized food recommendations.

For healthcare professionals, user-authorized food records could offer additional context when discussing dietary habits with patients.

Most importantly, our focus on regional food recognition makes MealSense relevant to communities whose everyday meals are often overlooked by mainstream nutrition applications.

What's Next?

MealSense is an evolving project, and we see several opportunities to improve it further:

  • Improve recognition accuracy across a wider variety of regional dishes.
  • Enhance portion-size estimation using computer vision and segmentation.
  • Improve nutritional estimation through verified food composition databases.
  • Expand personalized recommendations based on individual dietary requirements.
  • Optimize AI inference for faster and more resource-efficient deployment.
  • Strengthen privacy controls and doctor-user collaboration.
  • Evaluate nutritional prediction accuracy against real-world food measurements.

Our long-term vision is to make MealSense an intelligent nutrition companion that understands not only what people eat, but also how their food choices relate to their nutritional goals.

MealSense is our attempt to turn everyday food photographs into meaningful nutritional insights, making healthier eating easier to understand and manage.

Built With

Share this project:

Updates

Submission history