ThinK

ThinK helps you understand your meals instantly with AI-powered calorie estimates.

Comment

About the Project

Inspiration

A task you must do multiple times a day should be as simple as possible. Tracking calories is time-consuming and tedious, requiring manual entry before every meal. I wanted a tool that makes it effortless for anyone with a phone to keep track of their health. Imagine just taking a photo, and AI handles the complex work: snap -> understand -> estimate calories -> reach your goals.

That idea grew into ThinK, an AI system that identifies food, predicts calories directly from an image, and uses that data to manage your daily intake and help you reach your health goals.

What We Learned

This project pushed me into areas I had never explored this deeply before:

Developing this project was an eye-opening journey, particularly since we had no prior mobile app experience; it was surprising to discover how easy it was to get started and how fast we could prototype directly on a phone. The machine learning challenges focused on building a robust calorie predictor. We learned how to use a log-scaled regression model. A significant achievement was structuring the ML pipeline and implementing the necessary logic in Dart so the entire system could run fully offline on the user's device. Finally, the project taught us deep insights into classification and the importance of manipulating data effectively to get the outputs we needed.

How the Project Was Built

The application itself was developed entirely in Flutter/Dart, allowing us to deploy to multiple platforms from a single codebase. All the core machine learning was developed and trained separately using PyTorch and Python.

Model Development and Training

Our calorie estimation system was built by modifying and training on a powerful image recognition backbone. This core model was designed to process the input photo and generate a raw prediction of the meal's caloric content. This process involved:

  • Training a regression model to predict the log-scaled calories. This mathematical approach helps stabilize predictions across the wide range of meal sizes and calorie counts, significantly improving accuracy.

Model Optimization for Mobile

To ensure the model could run efficiently and natively on a user's phone, we converted our final PyTorch model to the ONNX format.

  • Since the native mobile environment (Dart) did not support all the model's complex post-processing steps, we had to manually re-implement the entire decoding pipeline in Dart to successfully load and execute the model on-device.
  • This optimization allows the entire prediction process to run fully offline.

The Combined Prediction System

ThinK goes beyond a single prediction by incorporating multiple factors to refine the calorie estimate.

  • The system uses priors based on the detected meal characteristics to adjust the raw prediction.
  • For example, meals identified as being calorie-dense receive a higher estimated multiplier, while those identified as being low in density receive a slight reduction.
  • This fusion of prediction and prior knowledge significantly improves overall accuracy without needing complex, manually labeled data.

Challenges

Machine Learning Pivot

We initially aimed for a complex object detection system, but faced significant hurdles trying to integrate models like YOLO after conversion. The process of converting the model to ONNX for mobile deployment yielded confusing and unusable raw outputs, forcing us to rebuild the entire decoding logic (like anchors and NMS) manually in Dart. Ultimately, due to time constraints, we were unable to train and finalize the object detection portion. This forced a pivot from classification (using dataset names) to a simpler regression model that purely focused on guessing the final calorie count from the image.

Dataset and Training Issues

Real-world food data is messy. Many samples had missing calorie labels or invalid paths, requiring heavy, time-consuming filtering before training could begin. Also, due to time constraints, we were unable to train a final, fully robust model for release.

Front-End and Deployment

Integrating the ML model into the mobile app was a precise challenge. ONNX Runtime is powerful, but it demands exact tensor shapes for input. Any mismatch immediately causes errors, requiring precise model introspection and strict data handling on the Dart side to ensure seamless, functional deployment. We had to focus significant effort on the front-end code to correctly handle the model's inputs and outputs.

Summary

ThinK makes health logging effortless by turning the tedious task of calorie counting into a simple photo snap. What started as a simple idea became a full offline computer-vision system built from the ground up. The application features a synth retro front-end design that gives it a nostalgic, poppy aesthetic.

The core technology uses a two-model pipeline that combines image recognition with a regression model to estimate calories directly from a photo. ThinK then uses that estimation to manage the user's daily intake and help them reach their health goals, all in a user experience that is fast, intuitive, and always available, running natively on a mobile device.

Built With

Share this project:

Updates