Chefly

Inspiration

Home cooks throw away food they forgot they had, and following a recipe on a phone or laptop with wet, floury hands is awkward: the screen dims, you lose your place, and you smear flour on the touchscreen. HackCamp asked us to identify a problem facing a community and build something that makes a real difference. Food waste in student and family kitchens felt like a loop worth closing: scan groceries → know what expires → cook a real recipe that uses it → touch nothing while cooking → pantry and nutrition update themselves.

What it does

Scan groceries. Point your camera at a barcode, or type it. Chefly looks it up through dataandapis.com, then Open Food Facts. If both miss, Gemini reads a photo of the product and its nutrition label, and you confirm every field before anything is saved.
Know what expires. Snap the printed best-before date. Gemini reads it, we cross-check it against the Canadian (CFIA) bilingual date format like 2026 NO 15, and you confirm. The pantry sorts itself by soonest expiry, with colour bands and plain-text labels ("Best before Nov 15 · 2 days").
Find real recipes. Gemini with Google Search grounding finds real recipe pages online, ranked to use whatever expires soonest. Each card shows how many ingredients you already have, which expiring items it uses, and calories and protein per serving. The original recipe is always shown first, credited to its publisher and author, with a "View original" link.
Adapt only when needed. If you're missing an ingredient, Chefly can suggest an AI-adapted version using what's in your pantry ("Missing: buttermilk → Using: Greek yogurt + lemon"). It appears in a separate, clearly badged card, and the original is never modified.
Cook hands-free. One big step per screen, the screen stays awake, and timers are parsed from the step text. Turn on hands-free and MediaPipe gesture recognition runs entirely in your browser: hold an open palm to arm, finger up and hand signs next/previous step, thumbs up/down to scroll, hold a fist to start/pause the timer, and hold a victory sign to have the step read aloud by ElevenLabs (with the browser's built-in voice as a fallback). Every gesture also has a big tap target and a keyboard shortcut.
Finish and log. "Finish cooking" previews exactly what will be deducted from your pantry, lets you edit it, then deducts ingredients and logs calories and protein for the servings you ate in one database transaction. The Today page shows kcal and protein rings against your goals and a 7-day history chart.

Diet and allergy filters (vegetarian, vegan, gluten-free, dairy-free, and Canada's priority allergens) are enforced in code on every ingredient line and every AI substitution, not just suggested in a prompt. Recipes that break a filter never appear.

How we built it

Next.js (App Router) + TypeScript, deployed as a standalone Node build in Docker on a Vultr VPS behind Caddy with automatic HTTPS. HTTPS isn't optional here: the browser only allows camera access in a secure context.
Auth0 Universal Login with Google. proxy.ts protects the app, and every API route and server action calls requireUser(), so the user id always comes from the session, never from the request. Each user only ever sees their own pantry, recipes and meals.
Tiger Data (TimescaleDB) is the only database. meal_logs and pantry_events are hypertables, a daily_nutrition continuous aggregate with a refresh policy and real-time aggregation powers the history chart, and the finish-cooking flow runs as a single transaction with an idempotency guard.
Gemini (gemini-3.8-flash) in four distinct roles: grounded search to discover recipe URLs, vision to read nutrition labels, vision to read expiry dates, and schema-constrained JSON for recipe adaptations, which the server validates before saving.
MediaPipe Tasks Vision gesture recognizer with a self-hosted WASM runtime and model, so camera frames never leave the device.
ElevenLabs eleven_flash_v2_5 streaming text-to-speech, proxied through our server so the key stays private, with speechSynthesis as the fallback.
Entire + Claude Code. Three Claude Code accounts built Chefly in parallel. A Phase 0 foundation froze the shared TypeScript contracts, SQL schema and stubs. Then three workstreams (Cook Mode, Pantry & Nutrition, Recipes & Cooking Backend) each owned separate directories, coordinated through Entire Sessions, Graph, Brain and Trails, before an integration phase merged them.

Challenges we ran into

Real recipes, not hallucinated ones. Combining Search grounding with JSON output has known bugs, and models sometimes invent URLs. So we split the job: Gemini only discovers URLs, we resolve Google's grounding redirects server-side, fetch the publisher's page through an SSRF-safe fetcher, and show a recipe only if the page returned 200 and parsed as schema.org Recipe markup. Results stream in progressively so the wait never feels dead.
Gestures that don't fire while you're chopping. We built an arm → hold → cooldown state machine, so random kitchen motion does nothing until you deliberately hold an open palm, and a single swipe can never skip two steps.
Timers that survive reloads and background tabs. Browsers throttle hidden-tab timers, so ours are stored as absolute end times rather than countdowns, and the alarm fires as soon as you come back to the tab.
Units. Recipes say "1 cup", pantries say "500 g". Everything is stored in base units (g, ml, count), and we only convert across dimensions when we know a density or per-piece weight. Otherwise we flag the line for the user instead of guessing.
Canadian date codes, where MA means May (not March), the year may be missing, and "packaged on" dates must never be treated as an expiry date.

Accomplishments that we're proud of

A full loop from barcode to plate to nutrition log that the user never has to touch while cooking.
Recipes that are real, attributed and shown as published, with AI used only to find and adapt, never to invent.
Strict, testable allergy enforcement in code.
A privacy-respecting camera: gesture recognition runs locally, and only photos you explicitly take for a label or date are sent for reading.

What we learned

How to design around LLM limitations: use the model where it's strong (search, reading messy labels) and let deterministic code handle what must be correct (allergens, unit maths, deductions).
TimescaleDB's hypertable rules (unique keys must include the time column, no hypertable-to-hypertable foreign keys) and how continuous aggregates make history queries cheap.
How to split one app across three AI coding agents by freezing contracts first, so each workstream can build against stubs without stepping on the others.

Community impact

Chefly targets household food waste by putting the items closest to expiry at the top of every recipe search. Hands-free gestures, read-aloud steps, large type and full keyboard support also make cooking easier for people with low vision, limited mobility, or simply messy hands. Chefly is not medical advice and always reminds users to check product labels for allergens.

What's next for Chefly

Voice commands alongside gestures.
Moving gesture inference to a Web Worker for smoother performance on low-end laptops.
"Similar recipes" search and smarter ingredient matching.
Pantry sharing for households and roommates.