Lil Gouda finna be snatched

TalkTuahNutritionist

Inspiration

Many nutritional assistance tools overlook individuals who face challenges in operating traditional devices, such as the elderly or those with neurocognitive conditions. We wanted to bridge this gap by creating a solution that improves accessibility to easy, personalized nutritional guidance.

Our app leverages AI-powered call agents, Meta smart glasses, and an onboard AI nutritionist to provide seamless assistance, ensuring that users who struggle with manual device interactions can still receive real-time, hands-free nutritional support effortlessly.

What it does

TalkTuahNutritionist is an accessibility-focused nutritional web app that makes personalized nutrition tracking effortless.

AI-Powered Setup: Users can call an AI agent to set up their system, which dynamically fills out their nutritional needs and goals based on weight and dietary preferences.
Real-Time Tracking: The app features a real-time nutritional tracker that adjusts recommendations as users progress.
Hands-Free Meal Analysis: Integration with Meta smart glasses allows users to capture food photos for automatic nutritional analysis.
AI-Generated Recipes: The system generates customized meal plans aligned with users’ health goals.
Entertaining Cooking Assistance: Recipes come with a Fetty Wap voice-over for step-by-step cooking directions, making meal prep engaging and fun.
Onboard AI Nutritionist: Users can engage with an AI-powered nutritionist that analyzes food logs, tracks nutritional goals, and provides real-time dietary advice through a retrieval-augmented generation (RAG) model.

With these features, TalkTuahNutritionist ensures hands-free, real-time nutritional tracking and guidance, making healthy eating more accessible.

How we built it

TalkTuahNutritionist was developed using a modern web stack combined with AI-powered accessibility features.

Tech Stack

Frontend & UI: Built with React, Node.js, and Tailwind CSS for an intuitive, responsive experience.
AI & Image Processing: OpenAI’s vision model powers food detection and analysis from Meta smart glasses.
Recipe Generation: Groq and Llama dynamically generate personalized recipes based on users' dietary needs.
Call Agent Automation: Retell AI enables users to speak with an AI-powered agent that automatically fills in their dietary preferences.
Hands-Free Interaction: Meta smart glasses capture food images, which are then processed using a web scraper to extract relevant nutritional data.
Voice-Guided Cooking: memeNome TTS transforms recipe instructions into an entertaining Fetty Wap voice-over.
Onboard AI Nutritionist: A retrieval-augmented generation (RAG) model analyzes user food logs, provides dietary feedback, and engages in real-time conversations about nutrition.

This combination of AI, automation, and accessibility tools ensures that users who struggle with traditional device interactions can still receive real-time, personalized nutritional guidance.

Challenges we ran into

Building TalkTuahNutritionist came with several technical and logistical hurdles:

Meta Smart Glasses Workaround: Since Meta does not provide a Developer Kit, we had to create a workaround. Instead of direct integration, we sent images to Messenger and then web scraped them for processing. This extra step added complexity but allowed us to keep the feature.
Fine-Tuning Vision Detection: Using OpenAI’s vision model for food detection was challenging. Fine-tuning LLMs required iterative testing due to misclassification issues and lack of proper segmentation in some images.
Call Agent Management: Retell AI, which powers our AI-based call setup, faced logistical errors in handling calls and user parameters. Ensuring smooth call processing and reliable user data capture required additional debugging and optimization.
Multi-AI System Integration: Coordinating multiple AI tools (OpenAI Vision, Groq/Llama for recipes, memeNome for TTS, and RAG for the AI nutritionist) required ensuring seamless compatibility, reducing latency, and resolving API mismatches.

Despite these challenges, we successfully built a fully functional system that brings AI-driven accessibility to real-time nutrition tracking.

Accomplishments that we're proud of

Successfully engineered a workaround for Meta Smart Glasses' lack of an SDK, enabling hands-free food logging by sending images to Messenger, scraping them, and processing them through an LLM API.
Integrated multiple AI models into a seamless system that delivers real-time, accessible, and engaging nutritional guidance.
Developed an AI-powered nutritionist that automates meal tracking, food log analysis, and personalized dietary recommendations.

Seeing the images correctly pass through the LLM API for analysis and interacting with the onboard AI nutritionist were breakthrough moments, proving that our approach worked despite the hardware limitations.

What we learned

Throughout this project, we gained valuable insights into AI integration, accessibility, and optimization:

Workarounds for Hardware Limitations: Since Meta Smart Glasses lack an SDK, we had to engineer a creative solution using Messenger and web scraping. This taught us how to adapt and work around hardware constraints effectively.
Enhancing AI Voice Agents: Working with Retell AI helped us refine voice AI automation, particularly in call handling, user intent recognition, and parameter processing.
LLM Optimization with Groq: We optimized LLM responses for faster, more accurate recipe generation, reinforcing the importance of fine-tuning AI models for practical use cases.
Implementing RAG for Nutrition Guidance: Developing an onboard AI nutritionist using a retrieval-augmented generation (RAG) model taught us how to create dynamic, user-specific AI responses for real-time health recommendations.

These lessons strengthened our technical skills and deepened our understanding of accessibility-driven AI solutions.

What's next for TalkTuahNutritionist

We're just getting started! Moving forward, we plan to expand TalkTuahNutritionist by:

Refining the AI Nutritionist’s Capabilities: Improving contextual responses and adding more personalized recommendations.
Expanding Multi-Device Support: Enhancing the smart glasses integration and exploring compatibility with more accessibility devices.
Optimizing Call Agent Efficiency: Improving call automation reliability and fine-tuning parameter handling for user preferences.

These enhancements will further improve accessibility, making nutrition tracking even more intuitive for users who struggle with traditional interfaces.

Our ultimate goal is to bridge the gap between AI-driven health recommendations and real-world accessibility needs, empowering users to take control of their nutrition effortlessly.

TalkTuahNutritionist isn’t just a tool—it’s a step toward inclusive, AI-powered nutrition tracking for everyone.

Eat well. Stay healthy. No screen required.