Nutrify Me

--
Inspiration

We wanted to tackle a real problem: most nutrition apps are just glorified calorie counters. They log
what you eat but never learn from you, never give you meaningful guidance, and certainly never adapt. We were inspired by the idea of building an actual AI nutritionist — one that remembers your preferences, understands your health goals, and gets smarter with every meal you log. Preventive health through
personalized nutrition, accessible to anyone.

What it does

NutritionMe is an AI-powered nutrition intelligence platform with three integrated agents that
share a semantic memory layer:

• 📸 Food Photo Analysis — Snap a photo of your meal and Claude (via Bedrock) instantly breaks it down: per-ingredient macros, a health score (1–100) grounded in preventive-health science, and actionable notes ("high fiber aids glucose control").
• 💬 AI Nutritionist Chat — A conversational nutritionist that knows your profile, remembers your past meals, and synthesizes long-term insights about your eating patterns. Responses are streamed in real time.
• 📅 Personalized Meal Planning — 7-day meal plans generated around your TDEE, dietary type, allergies, and health goals. Plans adapt if you eat outside them.
• 📊 Health Dashboard & Food Log — Day-by-day tracking with macro breakdowns, health score trends, and weekly averages.

The key differentiator: all three agents share one semantic memory layer — every interaction is embedded and retrieved via RAG, so the app genuinely gets smarter the more you use it.

How we built it

┌───────────────┬────────────────────────────────────────────────────────┐ │ Layer │ Tech │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Frontend │ Next.js 15, React 19, TypeScript, Tailwind CSS, Vercel │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Backend │ FastAPI (Python), AWS Lambda (ECR), Mangum │ ├───────────────┼────────────────────────────────────────────────────────┤ │ LLM │ AWS Bedrock — Claude Sonnet 4.5 │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Embeddings │ Amazon Titan Text Embeddings V2 (1024-dim) │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Vector DB │ Supabase PostgreSQL + pgvector (IVFFlat indexing) │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Auth │ Supabase Auth (email + Google OAuth) │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Async storage │ DynamoDB for food log, Aurora PostgreSQL fallback │ ├───────────────┼────────────────────────────────────────────────────────┤ │ Monorepo │ pnpm workspaces + Turborepo │ └───────────────┴────────────────────────────────────────────────────────┘

The architecture is fully serverless. Next.js on Vercel invokes Lambda directly (bypassing API Gateway's 30s limit) for streaming chat and meal plan generation. All AI logic lives in Python on Lambda; all user data reads live in Next.js with Supabase RLS ensuring users only ever see their own data.

The shared memory layer works like this: every meal analysis, every chat message, and every generated meal is embedded with Titan and stored in Supabase. At query time, three RPC functions (match_health_entries, match_messages, match_user_insights) retrieve semantically relevant context via cosine similarity, which is injected into the agent's system prompt before each response. Every 10 messages, Claude synthesizes 1–3 durable insights ("User prefers high-protein breakfasts") stored in a user_insights table — the app's long-term memory.

Challenges we ran into

• API Gateway's 30s timeout — Meal plan generation easily exceeds 30 seconds. We bypassed it by invoking Lambda directly from Next.js using the AWS SDK, which has no timeout ceiling.
• Aurora PostgreSQL on Lambda cold starts — Connection pooling in a serverless environment with IAM auth was tricky. We implemented lazy DB initialization and fallback to Supabase if Aurora is unreachable, so the app degrades gracefully rather than crashing.
• Streaming SSE from Lambda through Vercel — Getting Server-Sent Events to flow correctly through the Lambda → Next.js API route → browser chain required careful handling of response headers and stream flushing.
• Vercel monorepo config — Getting pnpm workspaces to build correctly on Vercel (root directory vs. install command vs. build command) consumed more time than expected.
• Keeping agent context concise — Injecting full meal history and meal plans into prompts hits token limits fast. We solved this by only injecting the last 20 meals and first 3 days of a plan, relying on semantic retrieval for the rest.

Accomplishments that we're proud of

• The shared semantic memory layer — Three independent AI agents that all "remember" the same user history, without any manual state synchronization. This is genuinely novel for a nutrition app.
• Adaptive meal planning — When you eat something off-plan, the system detects the deviation and can regenerate the remaining days around what you actually ate.
• Full serverless stack with real-time streaming — Chat responses stream word-by-word from a Python Lambda through a Next.js route to the browser, with no dropped frames or timeout issues.
• Preventive-health framing — Health scores and meal recommendations are grounded in disease prevention science (T2 diabetes, CVD, inflammation), not just "calories in / calories out".

What we learned

• pgvector in Supabase is production-ready — IVFFlat approximate nearest-neighbor search at millisecond latency, directly in Postgres, with no extra infrastructure. Extremely powerful for RAG.
• Serverless + streaming is hard but solvable — Lambda streaming mode + proper SSE headers + Next.js ReadableStream is a viable pattern once you understand the constraints.
• Long-term memory synthesis changes the product — Having Claude periodically distill durable insights from conversations (rather than just doing full-context retrieval) is what makes the nutritionist feel like it knows you rather than just searching your history.
• Graceful degradation matters in hackathons — Multi-database fallback logic (Aurora → Supabase → empty) saved us multiple times when cloud infra was flaky under time pressure.

What's next for NutrifyMe

• Mobile app (React Native) with native camera integration for even faster photo analysis
• Wearable integration — sync activity data from Apple Health / Fitbit to auto-adjust TDEE and meal plan targets
• Grocery list → delivery — one-click ordering from the generated grocery list via Instacart or similar
• Social & accountability features — share meal plans, challenge friends, compare health scores
• Deeper medical integration — connect with blood glucose monitors, cholesterol tests, etc. to close the loop between food choices and measurable health outcomes
• Fine-tuned embedding model — train a nutrition-domain-specific embedding model for more precise semantic retrieval

Built With

• amazon-web-services
• nextjs
• python
• vercel