Inspiration

PKU (Phenylketonuria) affects about 1 in 10,000 newborns worldwide. Patients must keep phenylalanine (Phe) intake under 200-500mg per day, which works out to roughly 4-10 grams of protein. Every meal is a risk calculation, and getting it wrong can cause neurological damage.

Most PKU families still track Phe with paper charts and memorized food lists. Mainstream nutrition apps don't know what Phe is, let alone exchanges or PKU safety thresholds. We wanted to build something that actually works for this community: point your camera at a meal, talk about what's in it, and get a Phe breakdown you can act on.

What it does

Talk to Gemini: Live Meal Analysis

The main feature: open your camera, show Gemini your meal, and talk through what you're eating. Gemini sees the food via live video, listens via audio, asks follow-up questions ("Is that regular bread or the low-protein kind?"), and responds by voice while building a Phe breakdown in real time. This runs on the Live API with function calling to extract structured nutrition data from the conversation.

Where this matters most:

  • Complex meals where a photo alone can't capture hidden ingredients. The back-and-forth conversation helps narrow things down.
  • Caregivers feeding infants who need hands-free interaction.
  • Situations where you want to ask "Can I have more of this?" and get an answer with context.

Photo Analysis with Gemini 3

For quick logging, snap a photo. Gemini 3 identifies each food item using Thinking Level: High for deeper nutritional reasoning, Google Search Grounding to look up Phe values from real sources, and high Media Resolution to catch portion details. Results come back as structured JSON with Phe in milligrams and safety levels (safe / caution / avoid).

When confidence is low, values are marked as estimates. In a health context, being upfront about uncertainty matters more than looking precise.

PKU Management Tools

  • Diagnosis OCR: Photograph a diagnosis letter. Gemini 3 extracts the prescribed Phe tolerance and sets up daily limits automatically, skipping manual entry.
  • Phe Dashboard: Color-coded progress bar (green to yellow to red) showing how much Phe allowance is left for the day.
  • Exchange Calculator: Converts between mg and PKU "exchange units" (1 exchange = 50mg Phe), the standard in PKU clinics.
  • Formula Tracking: Logs PKU medical formula intake with configurable time slots and reminders.
  • Blood Level History: Tracks Phe blood test results over time with trend charts and test reminders.
  • Caregiver Mode: Parents can link to their child's account, see intake remotely, and get alerts when Phe is getting close to the limit.
  • Multilingual: English, Korean, and Russian.

How we built it

Architecture

[Live Video + Audio] → Gemini Live API → Voice Conversation + Function Calling → FoodItem[]
[Photo / Barcode / Diagnosis] → Next.js API Route → Gemini 3 (Thinking + Grounding) → Structured JSON
                                                                                          ↓
                                                  Guest: localStorage  ←  App  →  Auth'd: Supabase DB

How We Use Gemini 3

Each Gemini 3 feature is tuned for a specific reason:

Feature Usage Rationale
Thinking Level Control high for multi-food meal analysis, low for barcode lookups Complex meals need reasoning chains (ingredient → protein % → Phe ratio → total mg). Barcodes just need a fast decode.
Google Search Grounding Real-time Phe lookup for regional and brand-specific foods PKU-specific products (low-protein bread, specialty formulas) aren't in static databases. Grounding finds them with source links.
Structured Output responseJsonSchema enforcing typed FoodItem[] with phe_mg, pku_safety, exchanges Health-related output can't be malformed. Schema enforcement means every response is valid and UI-ready.
Media Resolution high for food photos, medium for barcodes High resolution picks up details (sesame seeds, sauce drizzle) that affect Phe estimates. Barcodes only need enough to decode.

Gemini Live API

The live feature connects the browser to Gemini over WebSocket:

  • Video: Camera → JPEG frames at 1 FPS → WebSocket
  • Audio: Mic → 16-bit PCM 16kHz → WebSocket. Speaker ← 24kHz PCM responses.
  • Security: Ephemeral tokens from our backend. The API key never reaches the client. Tokens expire in 5 minutes, single-use.
  • Data Extraction: Function Calling (log_pku_food_analysis) pulls structured FoodItem[] out of the conversation as it happens.
  • Sessions: 2-minute limit with a visible countdown, auto-save on expiry, option to restart.

The system prompt tells Gemini to focus on PKU-relevant questions: what's in this dish, how was it prepared, what brand is that. It reasons about Phe from protein composition while flagging when it's estimating vs. citing a source.

Responsible Design

MyPKU is a decision-support tool, not medical advice. That shapes the whole product:

  • When Search Grounding provides Phe data, we show the source so users can verify it.
  • Estimates without grounding are labeled as approximations with conservative safety ratings.
  • When data is unavailable, the app suggests checking the label or asking a dietitian instead of guessing.
  • Every page includes a disclaimer: MyPKU supports PKU management but doesn't replace professional guidance.

Privacy

  • Ephemeral tokens: Live API uses single-use, time-limited tokens issued server-side. No API keys in the browser.
  • Row-Level Security: Every Supabase table uses RLS with auth.uid(). Caregivers access linked patients' data only through an explicit invite/accept flow with granular permissions.
  • Camera and audio streams are processed in real-time and not stored on our servers. Results save only to the user's own account.
  • Guest mode: you can use the app without creating an account.

Tech Stack

  • AI: Gemini 3 (gemini-3-flash-preview) + Gemini Live API
  • Frontend: Next.js 15 (App Router), React 19, TypeScript, Tailwind CSS
  • Backend: Next.js API Routes, Supabase (Auth, PostgreSQL, RLS)
  • State: Zustand with localStorage persistence (works offline, syncs when online)
  • PWA: Serwist service worker. AI calls: NetworkFirst. Food DB: CacheFirst with 90-day TTL.
  • i18n: next-intl (EN, KO, RU)
  • Auth: Google OAuth via Supabase, plus guest mode

PKU Food Database

We built a layered lookup to maximize coverage:

  1. Local PKU-specific database with verified Phe values
  2. Gemini 3 with Google Search Grounding for real-time lookup with sources
  3. USDA FoodData Central API
  4. Korean Food Safety API for regional foods
  5. Barcode scanning with cached results

Verified results get cached in Supabase, so every lookup makes the database better for the next user.

Challenges we ran into

Keeping live sessions focused: The hardest part of the Live API integration was making sure conversations produce usable Phe data, not just small talk. We constrained output through function declarations that enforce valid FoodItem structures, and wrote system instructions that steer Gemini toward PKU-relevant follow-ups ("What brand of flour is that?") instead of generic responses.

Thinking depth vs. speed: Gemini 3's Thinking Level adds accuracy but also latency. After testing different scenarios, we settled on high for multi-ingredient meals (where Phe estimation involves chained reasoning) and low for barcode lookups. The tradeoff felt right: take the time to think when accuracy matters, skip it when speed matters.

Phe data is hard to find: Standard food databases cover calories and protein well, but phenylalanine values are sparse. Search Grounding helped a lot here. Gemini can look up brand-specific products and regional foods that don't appear in static databases. For foods without any reference data, we fall back to estimating Phe at ~3-5% of protein by weight (a biochemical approximation) and mark those values clearly.

Working offline: PKU patients check food safety everywhere, including places without signal. We built a dual-storage system where Zustand syncs between localStorage (offline) and Supabase (online), with automatic migration when users sign in. The PWA caches the food database for offline barcode lookups.

Caregiver access control: The caregiver system needed Row-Level Security that lets parents see their child's data without giving up the child's control. We built an invite/accept flow with separate permission flags (view intake, view blood levels, receive alerts) enforced at the database level.

Medical terms across languages: PKU terminology doesn't translate cleanly. "Exchange units" mean different things in different clinical traditions. Getting Korean and Russian localizations right required understanding each region's medical conventions, not just translating words.

Accomplishments that we're proud of

  • The Live API conversation actually works for food analysis. A user can hold up a plate of bibimbap, say "the rice is about 200 grams, and I skipped the egg today," and get a Phe breakdown that accounts for both the visual and spoken context. Getting that loop to produce structured, usable nutrition data (not just chat) took a lot of iteration on prompts and function declarations.
  • Thinking Level tuning paid off. We can show a side-by-side where high correctly identifies that tofu skin has far more Phe than silken tofu, while low mode misses it. For a patient with a 400mg daily budget, that difference matters.
  • Search Grounding solved our biggest data problem. PKU-specific products like low-protein bread or regional formula brands don't exist in USDA or any static database. Grounding finds them with source links, and we cache the results so the next user benefits too.
  • The app works without an account and without internet. A parent at a grocery store with no signal can scan a barcode and get cached Phe data. No sign-up wall, no loading spinner.
  • Three languages (EN, KO, RU) with medically accurate PKU terminology in each. Not machine-translated labels, but terms that match what clinicians in each country actually use.
  • The caregiver system respects patient autonomy. Parents can monitor their child's intake, but the child (or their future adult self) controls who has access and what they can see. Permissions are enforced at the database level with RLS, not just in the UI.

What we learned

  • Thinking Level control makes a real difference for nutrition analysis. High thinking catches edge cases that fast mode misses. That kind of detail matters when your daily budget is 400mg.
  • Search Grounding fills a gap that static databases can't. PKU-specific products simply aren't in USDA or similar datasets. Grounding finds current data with sources.
  • Live conversation produces better results than single-shot analysis. Users can correct portions, clarify ingredients, and ask follow-up questions. The iterative process leads to more accurate Phe numbers.
  • Being upfront about uncertainty builds trust. Labeling estimates as estimates and showing sources makes people more comfortable using the tool, not less.
  • The PKU community (~50,000 patients worldwide) has very few digital tools built for them. Even straightforward features like a Phe progress bar or exchange calculator fill a real gap.
  • Structured Output combined with Function Calling creates a reliable data pipeline. In a health context, you can't afford malformed JSON.

What's next for MyPKU

  • Phe Correlation: Connecting dietary patterns with blood test results to surface personalized insights ("Your Phe tends to spike after meals with dairy").
  • Proactive Guidance: Using meal history to offer timely suggestions ("You're at 380mg of 400mg today. Here are some low-Phe dinner options").
  • Weekly Summaries: AI-generated weekly reports analyzing trends and highlighting what went well.
  • Community Recipes: A shared library of low-Phe recipes with verified nutritional data.
  • PKU Food Database Expansion: Continuously growing a verified, community-cached Phe database by combining Search Grounding lookups, public food APIs, and user-contributed corrections into a reliable shared resource.
  • Wearable Integration: Connecting with health devices for a fuller picture of PKU management.

Btw, I built it for my lovely girl and her PKU community!

Built With

Share this project:

Updates