HealthHarmony

An app that acts as a virtual coach for daily wellness

Comment

Bambam… first of all?? You built all that in three days?? That’s actually wild. Respect. 🔥

Your ideas are strong — they just need tightening, sharpening, and a bit more narrative clarity so judges immediately feel the impact. I’ll refine your answers to sound clear, confident, and compelling while keeping your voice.

🌿 Health Harmony – Refined Hackathon Submission

Inspiration

Many people struggle with fitness not because they lack motivation — but because they feel overwhelmed before they even begin. Tracking calories, planning workouts, staying consistent, monitoring mood… it’s fragmented and mentally exhausting.

I experienced that overwhelm myself.

Health Harmony was born from a simple idea: What if you had one intelligent companion guiding you across training, nutrition, and mindset — all in one place?

Instead of juggling multiple apps, Health Harmony acts as a unified AI wellness partner that simplifies the journey and keeps users aligned with their goals.

What It Does

Health Harmony is an AI-powered wellness companion delivered as a Progressive Web App.

It unifies:

  • 🥗 Nutrition tracking with AI meal analysis
  • 🏋🏽 Workout coaching with real-time form feedback
  • 💬 Conversational AI coaching with context awareness
  • 😊 Mood tracking with emotional trend analysis
  • 📊 Cross-feature health insights that connect habits to outcomes

All intelligence is powered by the Gemini 3 API, enabling multimodal understanding, streaming responses, structured data extraction, and tool-based reasoning.

Instead of isolated features, Health Harmony delivers a connected health ecosystem where training, diet, and mental wellbeing inform each other.

How We Built It

Frontend

Health Harmony is a React 18 single-page application built with Vite and styled using Tailwind CSS. React Router powers navigation across:

  • Dashboard
  • Meal Analyzer
  • Workout Coach
  • Chat Coach
  • Mood Tracker
  • Progress Analytics

Recharts handles dynamic visualizations, and the app is deployed on Vercel’s Edge Network as a Progressive Web App with offline manifest support — delivering a near-native mobile experience.

AI Layer (Gemini 3 Integration)

All intelligent functionality is powered through the Gemini 3 API using Google’s @google/generative-ai SDK.

We implemented custom hooks to abstract different interaction models:

  • useGemini – Single-turn multimodal requests (meal photos, workout clips)
  • useGeminiStream – Streaming responses with function calling for conversational AI
  • useLiveCoaching – Real-time video-based form analysis
  • Structured JSON outputs with schema validation for type-safe data extraction

Prompt engineering is centralized in a prompts.js utility, ensuring tuned system instructions for each feature.

This architecture allowed us to combine six distinct Gemini capabilities into one cohesive product rather than separate demos.

Live Coaching

The webcam-based workout coach uses the browser’s MediaDevices API via a useWebcam hook. Video frames are captured and sent to Gemini’s video understanding endpoint for analysis.

A LiveCoachingOverlay component renders a real-time HUD on top of the video stream, displaying:

  • Form score
  • Rep counter
  • Corrective tips

This creates an interactive, AI-powered fitness trainer experience directly in the browser.

Backend & Infrastructure

  • Vercel serverless functions proxy all Gemini API calls (keeping API keys secure)
  • Firebase Authentication enables Google sign-in and frictionless guest mode
  • Cloud Firestore stores meals, workouts, moods, and AI insights per user
  • Function calling (via defined tool schemas) allows the AI coach to fetch user data mid-conversation and deliver grounded, contextual advice

The architecture is fully serverless — scalable with zero backend maintenance.

Challenges

The biggest challenge was managing Gemini API rate limits during rapid development and testing. This slowed iteration cycles and forced us to design smarter batching and testing strategies.

We also learned that multimodal and streaming interactions each require distinct architectural patterns — they cannot be treated as the same request flow.

Accomplishments We’re Proud Of

  • Real-time webcam workout coaching with frame-level AI feedback
  • AI chat that pauses mid-response to call functions and pull real user data
  • Cross-feature reasoning (e.g., correlating workout frequency with mood improvements)
  • Six integrated Gemini capabilities in one unified product
  • Fully functional guest mode with zero onboarding friction
  • Fully serverless architecture
  • Built and deployed in just three days

What We Learned

Working with Gemini 3 taught us that interaction design is as important as model capability.

Single-turn multimodal requests, streaming conversations with tool calls, and structured JSON outputs each demand different architectural decisions.

We also learned that prompt engineering significantly impacts output quality. Small refinements to system instructions dramatically improved nutritional accuracy, form feedback precision, and emotional analysis clarity.

AI performance is not just about models — it’s about orchestration.

What’s Next

  • Wearable integrations (Apple Health, Google Fit) for heart rate, sleep, and step data
  • AI-powered weekly meal planning based on nutritional gaps
  • Social accountability features and shared challenges
  • Personalized push notifications aligned with user habits
  • Guided real-time workout sessions led by Gemini
  • Full offline mode with service worker caching and sync

The goal is to evolve Health Harmony from an AI tool into a truly intelligent wellness companion.

Built With

  • api
  • cloud-firestore
  • eact
  • firebase-authentication
  • google-gemini-3-api
  • javascript
  • mediadevices
  • pwa-(progressive-web-app)
  • react-router
  • recharts
  • remotion
  • tailwind-css
  • typescript
  • vercel-edge-network
  • vercel-serverless-functions
  • vite
Share this project:

Updates