FoodSense

FoodSense is a web app that uses real-time video on PC or mobile to track your meals. As you add or remove food, it instantly adjusts to give you the perfect portion and accurate nutritional info.

Comment

Inspiration

In today's busy world, we recognized a crucial need: leveraging technology to empower people in managing their nutrition effectively and efficiently, especially for homemade meals. Our health is our most valuable asset, yet it's often challenging to keep track of what we eat, particularly when we're cooking at home, eating at a restaurant, and/or when don't have access to nutritional labels.

We set out to solve two common problems faced by many, especially fitness enthusiasts and home cooks:

1-Accurately tracking the nutritional content of homemade meals quickly and conveniently 2-Understanding proper portion sizes without the hassle of weighing every ingredient

Calculating macros and estimating portions for every meal can be time-consuming and impractical. That's where FoodSense comes in—an innovative web app designed for speed and convenience. It scans your food, provides an instant nutritional breakdown, and helps you understand appropriate portion sizes, all without interrupting your routine. What sets FoodSense apart is its convenience. Being a web app, you can simply open your laptop or desktop computer and scan what you're eating without needing to get up or reach for your phone. This is perfect for those working from home or having meals at their desk.

What it does

FoodSense uses an OpenAI Vision Model and your phone or webcam to scan the food you're preparing, providing real-time feedback on the macronutrients (protein, carbs, fats) and caloric content. It warns you if you're exceeding your macros and helps you stay on track with your fitness goals. After each scan, you receive personalized feedback. The app has the potential to be a complete fitness planner and tracker powered by AI.

How we built it

We built FoodSense using a Next.js frontend connected to a custom backend through WebSocket technology, prioritizing user experience and performance. The frontend captures a video stream from the user’s device and sends it to the backend. From there, the backend intelligently selects and processes specific frames before passing them to an OpenAI Vision Model for analysis. Once the model processes the frames, it returns a JSON response with detailed information on macronutrients and calories, which is then displayed in real-time on the frontend. To further enhance the user experience, we added a feature where, upon completing a scan, users can receive personalized guidance from their AI coach, providing detailed ingredient information and suggestions based on what they’ve consumed.

Challenges we ran into

Integrating a performant WebSocket video stream connection was challenging at first. We encountered difficulties with parallel processing and thread management of the video streams, but through persistent problem-solving, we were able to resolve these issues. A major challenge was ensuring the Vision model could accurately recognize various foods and ingredients under different lighting conditions and angles. Additionally, designing a native app-like interface for the web platform across different browsers, while ensuring smooth access to the camera on multiple devices, proved difficult. However, we overcame these obstacles, and the end result is a cross-platform solution that combines the benefits of accessibility with a user-friendly design.

Accomplishments that we're proud of

We're incredibly proud of getting the app to function smoothly within the tight constraints of the hackathon. The real-time macro detection works with great performance, and we’re excited about how quickly we were able to integrate various technologies to create a seamless user experience. We're also proud of our teamwork—despite the fact that all team members met on Slack without prior personal connections, we quickly adapted to each other’s strengths and work styles, allowing us to collaborate effectively and bring the project to life.

What we learned

We learned a great deal about building fast, user-friendly apps while improving our understanding of networking, WebSocket technology, and parallel processing. Integrating AI through vision models gave us valuable experience in applying advanced technology to real-world scenarios. We also honed our teamwork and communication skills, quickly adapting to each other's strengths and collaborating efficiently, despite having just met online. Additionally, we focused on balancing functionality with user experience, ensuring the app is both efficient and intuitive. Hosting and deploying real-time applications further enhanced our ability to manage performance across platforms.

What's next for FoodSense

We believe that FoodSense has the potential to be a game changer. The platform can be greatly expanded to include tracking features and provide valuable insights into how society consumes food. The vast number of images captured by the app can also contribute to improving machine vision models through continuous training. We genuinely believe that with a few more weeks of work, this product is shipable and could have a significant impact on the world, especially in making calorie counting more accurate and accessible.

Given its access to real-time video, FoodSense could even evolve to assist with creating recipes and guiding users through the cooking process with its machine vision capabilities. While we successfully delivered the core functionality during the hackathon, future improvements could include offering personalized meal plans and generating weekly nutrition summaries to help users stay on track with their fitness goals.

Built With

Share this project:

Updates