Inspiration
The spark came from a single statistic: less than 15% of patients follow dietary advice from their doctor. Not because they don't want to — but because "eat more fiber" means nothing when you're standing in a corner store in the South Bronx with $180 on your SNAP card, three kids at home, and a cuisine that no mainstream nutrition app has ever heard of. We kept asking the same question: what actually happens between the doctor saying "eat better" and Tuesday's dinner? The answer was nothing. No tool, no bridge, no translation layer between clinical science and a real family's real kitchen. That gap is where diabetes gets worse. Where hypertension goes unmanaged. Where a pregnant woman doesn't get her folate because nobody told her that black beans and a squeeze of lime is a prescription, not just a side dish. We also noticed that the White House Conference on Hunger and Nutrition — the first in 50 years — had just made Food is Medicine a national priority. Kaiser, Mount Sinai, and Blue Cross were all running Produce Prescription pilots. The policy infrastructure was being built. But the consumer tool that actually delivers on that promise, for the communities that need it most, didn't exist. We decided to build it.
What it does
NutriRx is an AI that treats food like a prescription — written for your exact medical conditions, your exact SNAP budget, and your exact cultural kitchen. You start by building your household profile: each family member's age, sex, medical conditions (diabetes, hypertension, pregnancy, iron deficiency, and seven others), and optional lab values like A1C or blood pressure readings. You add your weekly SNAP balance, WIC eligibility, and describe your cuisine in plain language — "Dominican and some Chinese" works just as well as a dropdown. Then a panel of specialist AI agents activates — one per condition in your household. You watch them deliberate in real time: the Diabetes agent declares its glycemic targets, the Hypertension agent pushes DASH protocol, the Pregnancy agent demands folate. When they conflict, an Orchestrator agent resolves it on screen: you see the disagreement, you see the resolution, you see the reasoning. It's not a black box — it's a panel of specialists negotiating over your family's health. The output is a 7-day meal plan where every meal carries a clinical badge showing which condition it targets. Real dish names from your actual cuisine — Habichuelas Guisadas, Dal Tadka, Hoppin' John — not "bean soup" or "lentil dish." A shopping list grouped by category, sized to your SNAP budget, with WIC-eligible items flagged. Every item comes with a plain-English explanation of why it's in the prescription. After shopping, you upload a photo of your grocery receipt. NutriRx OCRs it, decodes store abbreviations, grades your adherence against the prescription, and gives you one specific swap for next week — same store, same price tier, meaningfully better nutrition. Next week's plan adjusts based on what you actually bought. Over time, the History page shows your clinical markers trending in the right direction week over week.
How we built it
The architecture has two layers that mirror how a real medical team works: specialists who think, and function agents who execute. The specialist layer is ten pre-defined AI agents — one per medical condition — each with a deep system prompt embedding the relevant clinical guidelines: ADA 2024 for diabetes, AHA DASH protocol for hypertension, ACOG prenatal standards for pregnancy, AAP guidelines for pediatric nutrition. Each agent takes the household profile and returns a structured JSON of nutrient targets, preferred foods, foods to avoid, and conflict flags. We run them in parallel using asyncio.gather and then pass all their outputs to a Gemini 1.5 Pro orchestration round that resolves conflicts and produces a single merged nutrient target vector. The function layer takes that merged vector and executes: the Cultural Agent generates 28 culturally-authentic meals via Gemini 1.5 Flash using the cuisine the family described in free text, then validates each ingredient against the USDA FoodData Central API for real nutrition data. The Budget Agent sizes everything to the SNAP balance using USDA Thrifty Food Plan price data. The Grading Agent receives OCR output from Google Cloud Vision, decodes abbreviations through a Gemini cleanup step, and scores adherence against the original prescription. The entire pipeline streams to the frontend via Server-Sent Events so the agent panel updates live as each agent completes. The frontend is Next.js 14 with Framer Motion driving the animated agent cards, Recharts for the nutrition radar and trend charts, and a design system we called Clinical Noir — dark and authoritative, but warm at the human moments. Every color in the UI is semantically meaningful: amber always means glycemic, rose always means iron, blue always means blood pressure. All data is live. No mocks anywhere in the codebase. USDA FoodData Central for every ingredient lookup. NIH Dietary Reference Intakes tables for age and sex-specific RDA targets. Google Cloud Vision for receipt OCR. Unsplash for real food photography per dish. SQLite stores plans and grades for the history and trend views.
Challenges we ran into
The negotiation round was harder than we expected. Getting Gemini to reliably produce valid JSON when resolving conflicts between five agents — some of which have genuinely contradictory recommendations, like Kidney Disease restricting potassium while Hypertension demands it — required significant prompt engineering. The conflict resolution prompt went through eleven iterations before it consistently produced resolutions that were both clinically sound and coherent enough to display on screen. Receipt OCR is messier in practice than in theory. Grocery receipts say things like GV WHL WHT BRD 2/$5.00, BNLESS CHKN THGH, and MED SHRIMP FRZ. The Vision API handles the characters fine; getting Gemini to reliably decode store abbreviations into USDA-matchable food names without hallucinating ingredients took a dedicated cleanup step and careful prompt design. We also had to handle receipts where the camera angle caused text to skew or where thermal paper had faded. Cultural recipe generation needs guardrails. Asking Gemini to generate authentic Dominican recipes that also hit a specific glycemic target and a potassium floor is a genuinely hard multi-objective prompt. Early versions produced dishes that were nutritionally accurate but culturally vague — "rice with beans" instead of "Arroz con Habichuelas Guisadas." We solved this by requiring the model to name specific dishes first, then validate ingredients, which anchored the output in real culinary tradition rather than generic nutrition. SSE streaming and async coordination in FastAPI had subtle race conditions when multiple agents completed near-simultaneously. The generator had to be carefully structured so that agent updates streamed in order and the final complete event always arrived last with the fully assembled plan. SNAP budget constraints are tighter than they look. Designing a 28-meal plan for a family of three at $180/week — that also hits therapeutic iron, folate, and glycemic targets — required the Budget Agent to do real substitution reasoning, not just cost-checking. Getting it to swap "fresh salmon" for "canned sardines" on clinical grounds (same omega-3 profile, one-eighth the cost) while explaining the substitution to the user was one of the most satisfying problems to solve.
Accomplishments that we're proud of
The thing we're most proud of isn't the technology — it's the moment during testing when we ran Maria's profile (the Dominican grandmother with diabetes, hypertension, and a pregnant daughter-in-law) and the plan came back with Habichuelas Guisadas on Tuesday, Pollo Guisado on Thursday, and a clinical rationale badge on each one. It looked like a prescription written by someone who actually knew her family. That felt like the whole point. We're proud that we shipped zero mock data. Every nutrition number in every meal card is a live USDA FoodData Central lookup. Every recipe is generated fresh by a model with genuine clinical constraints. Every receipt grade comes from a real Vision API OCR pass. In a hackathon, the temptation to fake the data layer is enormous. We didn't. We're proud of the agent negotiation panel — specifically the conflict banner. The moment where the screen shows "DiabetesAgent wants low carb → PregnancyAgent needs folate-rich whole grains → Resolution: use low-GI whole grains like quinoa" is the moment that explains why multi-agent architecture exists. It made the AI's reasoning visible in a way that felt genuinely useful, not theatrical. We're proud that the app works for any cuisine the user types in free text. "Oaxacan and some Korean" generates real Oaxacan and Korean dishes with clinical validation. The free-text cuisine field was a small product decision with enormous equity implications — it means the app doesn't require communities to fit into a predefined dropdown. And we're proud of the receipt loop. The swap suggestion — one specific, same-store, same-price-tier trade that closes a real clinical gap — is the hardest thing to get right and the most meaningful output in the whole system.
What we learned
We learned that multi-agent systems are most valuable at the negotiation layer, not the execution layer. The Cultural and Budget agents could have been single prompt calls. The specialist agents could have been one big prompt with condition-specific sections. What genuinely requires agents is the conflict resolution — when Kidney Disease and Hypertension have incompatible requirements, you need separate voices that can be heard and adjudicated, not a single prompt trying to hold contradictory clinical guidelines in its context simultaneously. We learned that clinical accuracy and cultural authenticity are not in tension — they reinforce each other. Traditional cuisines from the Dominican Republic, Bangladesh, Ethiopia, and the American South are built around legumes, vegetables, whole grains, and lean proteins. They are, in many cases, closer to therapeutic nutrition patterns than the processed food landscape they've been displaced by. The Cultural Agent's job isn't to make healthy food "culturally acceptable" — it's to recognize that the cultural food often already is the healthy food, with small adjustments. We learned that the receipt grading loop is the feature that turns NutriRx from a one-time generator into a system that actually changes behavior. Without it, the app produces a plan and the user's relationship with it ends at the shopping list. With it, the loop closes: behavior is measured, the gap between intent and action becomes visible, and next week's plan gets smarter. That feedback mechanism is the real product. We also learned, more practically, that prompt engineering for structured JSON output from Gemini requires extreme specificity in the output schema. Vague instructions produce creative interpretations. The prompts that worked best were the ones that gave an exact JSON template with field names, types, and a filled example — leaving the model nothing to invent except the clinical content.
What's next for NutriRx
The most important next step is clinical validation. We'd want to partner with a community health center — ideally in a food-insecure urban neighborhood — to run a 12-week pilot with real SNAP families and track whether NutriRx adherence correlates with measurable A1C improvement, blood pressure reduction, or iron level normalization. Without that validation, NutriRx is a well-designed tool. With it, it becomes evidence. On the product side: voice input for families who find typing difficult or for whom English is a second language. A Spanish-first interface for the demographic most likely to use this in New York City. Integration with WIC benefit systems so the app can show which specific WIC vouchers cover which items at which stores. A community mode where a neighborhood health worker inputs a whole block's worth of households and generates a bulk seed recommendation for a community garden or food pantry. On the clinical side: a pathway to connect NutriRx-generated plans to actual produce prescription programs — so a doctor can write a NutriRx and the patient receives a subsidy or voucher for the specific items in their plan. The infrastructure for this exists through programs like DC Central Kitchen's Healthy Corners and SNAP incentive programs like Double Up Food Bucks. NutriRx could be the digital layer that makes those programs personalized rather than generic. Longer term: a provider-facing dashboard. Community health workers, WIC nutritionists, and clinical dietitians should be able to see their patients' adherence trends, flag weeks where clinical targets are consistently missed, and adjust the prescription — the same way a doctor adjusts a medication dose. That's what Food as Medicine actually means in practice, and that's the system NutriRx is trying to become.
Log in or sign up for Devpost to join the conversation.