Dietician Assisting Multi-Model Agent

Cloud Run Logs to prove backend runs in GCP
Rag Database in Firestore
home page

Inspiration

The spark for NutriBuddy came from seeing the daily grind of a teammate's cousin, a professional dietician. We noticed a frustrating pattern: follow up appointments were often consumed by repetitive data gathering, asking the same questions about meal logs and habits, leaving almost no time for actual nutritional coaching. We realized that by automating these high frequency, low variance interactions, we could slash the workload for dieticians and, by extension, make personalized nutrition more affordable and accessible for everyone.

What it does

NutriBuddy acts as an intelligent, real time bridge between dieticians and their patients. It features a Gemini Live agent that conducts bidirectional voice and visual conversations. Patients can literally show their meals to the camera or chat about their day.

The agent is not just chatting into a void; it has full context via a RAG (Retrieval Augmented Generation) system containing the patient's medical history and the dietician's specific protocols. At the end of a session, the agent logs the data into a structured format and generates a concise summary, allowing the dietician to review days of progress in just a few seconds before a formal check in.

How we built it

Our stack is built for speed and scalability:

Backend and Frontend: Developed using Python and JavaScript.
Infrastructure: Deployed on Google Cloud Run for serverless efficiency.
Storage and Memory: We used Firestore as both our primary database and our vector store for RAG, ensuring seamless data retrieval.
Object Storage: Google Cloud Storage handles larger files and persistent log archives.

Challenges we ran into

The learning curve for bidirectional streaming was steep. Synchronizing real time audio and video with Gemini's processing while maintaining low latency was a puzzle we had not tackled before. We spent significant time debugging websocket stutters and ensuring the model did not hallucinate over the patient's records. Huge thanks to the antigravity; it was a lifesaver in helping us navigate the complexities of building this system.

Accomplishments that we are proud of

We are incredibly proud of the low latency real feel conversation. Achieving an interweaving of RAG and a live agent that responds in near real time was a major technical win. Beyond the code, we are proud that we built a personality for the agent that is not clinical or robotic; it is engaging, which is crucial for patient retention in a healthcare context.

What we learned

This project was a deep dive into the world of WebSockets and real time data flow. We moved beyond basic REST APIs and learned how to manage persistent, stateful connections. We also gained hands on experience in cloud architecture, specifically learning how to leverage Firestore as a lightweight yet powerful vector database for RAG.

What is next for NutriBuddy

The journey does not end here. Our roadmap includes:

WebRTC Integration: To further reduce latency and provide a more robust experience for users on unstable mobile connections.
Advanced RAG Pipeline: Implementing document importance weighting and automated deletion for outdated records to keep the model's context sharp.
Pilot Program: We are currently preparing to pitch NutriBuddy to local clinics to gather real world feedback and if possible provide them this as a paid solution.