Inspiration
The Meta Quest 3's passthrough camera represents a significant technological breakthrough, providing users with the ability to seamlessly integrate their virtual experience with the real world. A substantial portion of Quest users leverage VR for fitness activities. However, a truly holistic approach to a healthy lifestyle necessitates addressing nutrition in addition to physical exercise which was lacking on the quest store. Hence, to deliver an end-to-end system that comprehensively elevates the user's fitness journey, I developed NutriSmart.
What it does
NutriSmart covers the complete lifecycle of healthy meal consumption, starting with the informed selection of ingredients, guiding the user through the cooking process, and culminating in a detailed analysis of the final cooked food.
1. Barcode Scanning & Analysis: The platform features an integrated Barcode Scanner that instantly analyzes ingredient labels, providing users with in-depth data on nutritional composition and lifestyle impact.
2. AI-Guided Meal Preparation: The Cook with AI module utilizes available ingredients to generate healthy recipes with clear, step-by-step instructions and audio narration.
3. The Food Analyzer: Food Analyzer allows users to assess their prepared meals, delivering a comprehensive breakdown of the dish's nutritional value and macro-nutrient profile.
How we built it
Barcode Reader: Nutrismart employs the YOLO computer vision model for the real-time detection of barcodes, outputting precise bounding box coordinates. The bounding box is cropped, upscaled, brightened and sent to Zxing library. If Zxing still fails after few tries, the processed image is routed to the Gemini API. The resulting barcode value is then utilized as a query for the Open Food Facts API. The retrieved nutritional data is synthesized, structurally organized, and presented to the user. Concurrently, the Gemini API's text-to-speech (TTS) capability is invoked in a different thread to provide a narrated output.
Cook with AI: User is requested to take an image of the ingredients. The image texture, paired with a customized instruction set, is transmitted to the Gemini API. The API returns a structured JSON object, which is then deserialized and parsed within the Unity 3D. The colors of the app is dynamically updated, organized and the data is shown to the users. The specific instructions for text to speech is sent to Gemini in the background.
Food Analyzer: - User is requested to take an image of the food. The image texture is converted to base64 and then sent to Gemini API along with specific set of customized instructions. The received JSON is parsed, organized and the information is shown to the users. Concurrently, Gemini's TTS is invoked in the background.
Challenges we ran into
The biggest challenge was to read the barcodes reliably. Barcodes on packaging are of different sizes and Zxing library is really old and can't reliably parse Meta camera's texture. Hence, I decided to use YOLO computer vision model to detect the barcodes. However, many image processing steps like (cropping, upscaling, image adjustments etc.) is implemented before sending the texture to Zxing. The enhanced pipeline ensures a high rate of successful local decoding via Zxing. The Gemini API is configured as the backup OCR service, engaging only after successive Zxing parsing failures.
Accomplishments that we're proud of
I am very proud that Nutrismart reliably reads smaller barcodes, handles low light and also reads barcodes on wrinkled or curved packaging making it a tool which can be used in most conditions.
What we learned
I learned a lot about Tensors in Unity, yolo AI models and how unity Sentis works. Further, I learned a lot about Mixed Reality and Meta's passthrough camera.
What's next for Nutrismart
Nutrismart is a very useful tool, however, one important improvement would be:
- Currently, Nutrismart uses Open food facts API. Even though it is a great database and contains information on most food products, it still misses some food products. Nutrismart needs a more reliable and premium database so it is able to show information on food products in every situation.
Log in or sign up for Devpost to join the conversation.