HappyDiet

• 

  Architecture

Inspiration

My doctor and nutrinist gave me a diet on a handwritten sheet and a blank template to fill in. As a software engineer, it was obvious why most diets fail: tracking manually is pure friction and difficult, especially when you’re hungry and you're tired.

Talking to nutritionists and patients confirmed my hypothesis. People don’t need another calorie counter, they need support, simplicity and doctors need data-evidence, in the moment, “Does this meal follow my doctor’s rules?” That’s the gap HappyDiet is built to close.

What it does?

HappyDiet is a voice-first, photo-based diet coach that measures real compliance with your diet, not just calories. You snap a picture, ask your coach, and get an instant, spoken verdict while you can still change your choice. You can have the control on your hand with an iOS app or with a web application. Both work interchangeably.

The system focuses on rules like “no gluten,” “max 2 coffees,” or “under 1,800 kcal, all this thanks to the Vision AI that helps to parse and do a great OCR but also a great step structuring data persistently into a database.

This rule engine become the basis that checks your meal against your personal rules, scores compliance, explains what was on-plan and off-plan, and suggests simple tweaks for next time, all that can be prompted to the coach agent.

How I've built it?

I was constrained on time but the idea was running in my head for months. So I've used Cursor to accelerate code and use the framework of spec-driven with strong TDD & version control into to avoid drift and keep code stable as far as I could.

Under the hood, HappyDiet follows a simple loop: See → Think → Speak.

See: Gemini 2.5 Flash Vision turns a meal photo into structured ingredients and calorie estimates. It also parses handwritten diet plans into machine-readable rules.

Think: A custom Rule Engine in the backend compares ingredients and calories against your rules stored in Firestore, using exact, fuzzy, and semantic matching, then computes a compliance score and calorie penalties.

Speak: Gemini Live powers a real-time, interruptible voice coach that explains the result in natural language. The whole system runs on Google Cloud (Cloud Run, Firestore, Vertex AI) with a tested backend and CI/CD.

There’s a web version on Cloud Run and a native iOS app with camera, photo library, Live Coach, and support for affordable HeyCyan smart glasses for hands-free capture. I wasn't able to finish totally a seamless integration but a little bit more effort will get there.

Challenges we ran into

The first prototype was one giant prompt, with no memory beyond the session, so manual work arounds to persist the data, stability of the rules came in place. None that worked fully under ChatGPT so I've abandoned that project.

So this time, I knew I had to be serious about the system architecture and rebuilding it into a three-agent pipeline with a pure-code Rule Engine took a full architecture rethink.

Accomplishments that I am proud of

HappyDiet doesn’t just tell you what you ate; it tells you whether you followed your specific diet rules. That’s a big shift from generic calorie logging.

I've also delivered a solid engineering: a clean Reader–Processor–Reporter pipeline, a deterministic Rule Engine with full test coverage, and a smooth Live API experience with barge-in. On top of that, the voice-first, photo-based flow makes it usable for people who hate traditional logging—elderly users, busy parents, or anyone who just wants to ask, “Am I on track?”

What I've learned

The key lesson: friction, not knowledge, is what kills diet adherence. Removing typing, weighing, searching your diet again to know if you are compliant, asissting you to buy your emails, those key moments of truth on the user journey make the full difference.

I've also learned that AI models are the easy part; the hard part is the glue—data modeling, state, error handling, and scoring that feels fair. Combining a strict Rule Engine with LLM semantic understanding turned out to be the right balance between control and flexibility.

What’s next for HappyDiet

Next, I want deeper tools for nutritionists: better dashboards, trends over time, and integrations with clinical workflows. On the user side, richer coaching and more personalized insights (patterns, triggers, suggestions) are on the roadmap, but already the coach can do so if prompted.

Moreover, that's why also wanted the lenses to get into the ecosystem: you can easily lose compliance if your users get bored of taking pictures of their meal every time. It happened to me.

Lastly, this can have a real impact on health outcomes, so worth to test this hypothesis beyond.

Pablo Miranda – Founder & CEO, HappyDiet Built for the Gemini Live Agent Challenge, running entirely on Google Cloud.

Built With

• cloudrun
• cursor
• firestore
• gemini
• ios
• liveapi
• node.js
• vertex
• web