MissingBite

MissingBite creates a private, confirmed kitchen snapshot from photos of any useful kitchen area, preserves a handwritten family recipe as both an original artifact and usable structured data.

Comment

Project Story

Inspiration

MissingBite began with a simple question during a live visual conversation: could AI look inside our fridge and tell us what we could cook?

The experiment revealed a more interesting problem. Blurry video, changing angles, overlapping items, and incomplete views made confident answers unreliable. A trustworthy kitchen assistant should not pretend it has seen everything. It should work from clear evidence, show uncertainty, and let the user confirm what is actually true.

Then Despoina added the idea that transformed the project: preserving handwritten family recipes. These pages carry memories and personality, but they are difficult to search, compare with current ingredients, or follow hands-free while cooking.

We realized we were not building another recipe generator. MissingBite would connect two sources of personal household knowledge: the food people actually have and the recipes they already love.

What it does

MissingBite creates a private, user-confirmed kitchen snapshot from photos of any useful kitchen area. A user can photograph a fridge shelf, freezer, pantry, cupboard, or another area without completing a rigid full-kitchen scan.

GPT-5.6 turns those photos into reviewable item candidates. Users can correct, reject, or confirm each result before anything enters inventory. Partial future scans refresh confirmed matches without adding uncertain visual quantities or deleting food that was outside the camera view.

Users can also import a handwritten recipe from up to eight ordered images. MissingBite preserves every original page as a private artifact while creating a consistent, editable Cook Mode with a title, ingredients, servings, and steps. Recipes can also be typed, pasted into the structured fields, or dictated.

Deterministic application logic—not another model guess—compares each recipe with confirmed inventory:

  • Have means a matching item is confirmed.
  • Check means a match exists, but its identity or quantity may need verification.
  • Missing remains explicitly missing.

Missing ingredients can be added to an editable shopping list in one reviewed batch. The recipe amount becomes the initial need, but users can change it to the package they intend to buy. Checking an item records shopping progress only. A separate reviewed transfer confirms what was actually purchased, where it belongs, and when it should enter inventory. Recipe availability then recalculates automatically without altering the recipe itself.

An optional food profile provides cautious, non-medical suggestions based on user-declared preferences and avoidances while keeping the complete cookbook visible.

MissingBite also includes a live kitchen companion. It can retrieve recipe ingredients, guide Cook Mode step by step, navigate the app, and propose inventory or shopping actions while the user continues cooking. Every durable voice action still pauses for visible approval.

How we built it

The original idea and early planning predated Build Week. During the event, Codex became our engineering collaborator, helping turn the product decisions into a working application through small, reviewable vertical slices.

Codex supported:

  • Product scoping, architecture, and UX design
  • Expo and FastAPI implementation
  • Strict AI schemas and deterministic state logic
  • Supabase migrations, row-level security, and private Storage
  • Automated tests, CI, deployment, and rendered QA
  • Security reviews, failure handling, documentation, and release hardening

The client uses Expo, React Native, and TypeScript. FastAPI provides protected AI and Realtime credential routes. Supabase supplies authentication, PostgreSQL persistence, row-level security, and private recipe-image storage. Render hosts the web client and API.

GPT-5.6 Terra powers live multimodal kitchen analysis and handwritten-recipe transcription through the OpenAI Responses API. Kitchen scans use high reasoning effort, while the more difficult handwriting workload uses xhigh reasoning and original image detail. The backend requires strict, versioned structured output, and every result enters the application as a candidate or draft—not as a confirmed fact.

Deterministic code owns matching, arithmetic, revisions, shopping operations, and state transitions.

Hands-free conversation uses gpt-realtime-2.1 over WebRTC with short-lived, server-minted credentials. The OpenAI project key never enters the client. Voice tools use strict schemas and client-enforced approvals, while household data remains protected by the same application boundaries as touch interactions.

Challenges we ran into

The hardest challenge was deciding what AI should not be allowed to decide.

Image recognition can miss an item, merge similar products, double-count overlapping photos, or misunderstand handwriting. Instead of hiding that uncertainty, we made it part of the product through evidence, unknown fields, editable drafts, and explicit confirmation.

Inventory was also more complicated than a simple list. Multiple photos must not multiply quantities. A partial rescan must not erase unseen food. A shopping checkmark must not imply that something is already in the kitchen. These cases required conservative merging, revision-aware persistence, and separate reviewed state transitions.

Preserving handwritten recipes introduced another tension: users needed a clean and editable version without losing the emotional value of the original pages. We solved that with separate immutable Original and editable Cook Mode views.

Voice brought its own challenges: kitchen noise, reconnects, stale recipe context, bulk approvals, mobile layout overlap, and keeping the application usable during a conversation. We addressed these with far-field noise reduction, scoped tool access, snapshot-bound approvals, lifecycle handling, and responsive testing.

We also had to balance GPT-5.6 recognition quality against latency. Higher-resolution images and greater reasoning substantially improved difficult handwriting, but naturally took longer.

Finally, we resisted expanding into nutrition databases, retailer checkout, universal recipe-website import, and other attractive ideas before the central experience was reliable.

Accomplishments that we're proud of

We are proud that MissingBite closes the complete loop:

Observe the kitchen → confirm inventory → preserve a personal recipe → understand the gap → shop deliberately → confirm what was bought → recalculate what can be cooked.

The confirmation boundary is real product behavior, not merely a disclaimer. AI output cannot silently become household truth.

The Original/Cook Mode design preserves the history of a family recipe while making it genuinely useful. Have, Check, and Missing give uncertainty a clear language instead of disguising it as confidence.

The live kitchen companion works across the application while respecting the same approval and state boundaries as the visual interface. Users can continue using MissingBite while talking, and voice cannot bypass confirmation for durable changes.

We also created a resettable Seeded demo that follows the same contracts without consuming OpenAI credits or pretending that its synthetic output came from a live model.

The project has grown beyond a prototype into a deployed application with private user-scoped persistence, protected AI routes, responsive QA, CI, and extensive automated coverage—including 331 mobile tests, 110 API tests, strict type and lint checks, and a production web export.

What we learned

We learned that the strongest AI product is not necessarily the most autonomous one. For household facts, “I am not certain—please check” can be more valuable than a confident invention.

We learned to preserve provenance. Structured data becomes more useful when the original source remains available and corrections never erase it.

We also learned that multimodal AI and deterministic software complement each other. GPT-5.6 is excellent at turning messy visual information into useful structure. Conventional code is better for matching, quantities, revisions, and irreversible state changes.

A shopping checkmark represents progress, not proof of purchase. Keeping shopping and inventory as separate states made the entire system more trustworthy.

Voice works best as another interface to the same application—not as a separate assistant with its own memory and rules. Shared state, bounded tools, and visible approvals made the conversation natural while keeping users in control.

Most importantly, a focused product with an honest fallback produced a stronger result than a wide collection of unfinished features.

What's next for MissingBite

Our first post-competition priorities are:

  • Signed Android and iOS releases with physical-device testing
  • Locale and measurement preferences, including metric, US, UK, and Australian cooking units
  • Production onboarding, account lifecycle, privacy-safe usage metering, entitlements, and billing
  • A ChatGPT App backed by a thin authenticated MCP layer, beginning with read-only access to confirmed inventory, recipes, matching, and shopping
  • Cited nutrition and product-label data, portion scaling, and reviewed substitutions
  • Meal planning based on confirmed inventory and explicit user goals
  • Shared household roles and invitations
  • Regional shopping and retailer integrations

Every future feature will preserve the principle established by the MVP: AI may propose and explain, but users confirm the facts that shape their kitchen.

Built With

Share this project:

Updates