Kitchen Copilot

A cooking assistant that plans several dishes for one cook and safely replans mid-cook without forgetting what you've already done, asking instead of guessing.

Comment

Inspiration

Cooking a few dishes at once, say packed lunches for three people, means interleaving steps, scaling quantities and keeping track of what's done. The hard moment is a change of plan halfway through: "actually, make them all vegetarian". By then some food is already prepared. A good helper has to keep finished work as it is (you can't un-marinate chicken), work out only the remaining changes, and know what it doesn't know. If raw chicken was already prepared, was it kept apart from the other food? Only the cook can answer that, and the answer decides whether a dish can honestly be called vegetarian. Chat assistants tend to guess here, and in our own live tests the model did exactly that.

What it does

  • Plans multiple dishes for one cook: curated recipes with explicit quantities, scaled to servings with kitchen-friendly rounding, and scheduled so hands-off steps (simmering, resting) overlap with hands-on ones.
  • Shows what to do now: an Up next card with the step, its quantities, safety notes and a Mark done button. Steps that depend on unfinished ones can't be completed.
  • Replans without rewriting history: completed steps keep their quantities and times. If prep was done for fewer servings, a prep top-up step covers the difference. A What changed summary counts new, updated, dropped and kept steps.
  • Asks before relying on unknown physical facts: "Was the prepared chicken kept separate, or mixed into another dish?" The question stays visible until answered and survives a refresh. Uncertain answers record nothing, and mixed food can never be relabelled vegetarian.
  • Enforces diets in code: dietary status comes from ingredient traits, not from labels a model writes.
  • Two assistants, clearly labelled: a deterministic demo (no AI model, no AWS) and a Bedrock mode where Amazon Nova 2 Lite chooses the tools from natural language.
  • Optional browser voice: tap-to-dictate fills the message box for review, and nothing is sent until you press Send. Read reply and Read next step read the on-screen text aloud on request. Typing always works.
  • Durable and honest: request ids, version checks and receipts mean a retry never applies a change twice. Errors and refusals are shown as "not applied", never as success.

How we built it

  • A real MCP server over Streamable HTTP (official MCP TypeScript SDK, protocol 2025-11-25) is the only way to change a plan. The app's own chat route connects to it with the official MCP client, exactly as an external agent would.
    • Tools: list_recipes, create_cooking_session, get_cooking_session, update_cooking_session (including a whole-order replan), complete_cooking_step, and the read-only preview_replan.
  • Bedrock mode uses the AWS SDK for JavaScript v3 with the Converse API and tool use, running Amazon Nova 2 Lite through the us.amazon.nova-2-lite-v1:0 inference profile in us-east-1.
  • The application keeps authority. Every model-selected tool call is validated before dispatch: strict schemas, ids and versions supplied by the app, and physical facts and completions that need the user's own words. The MCP server re-validates everything.
  • Semantic replan with a typed question. "Three vegetarian lunches" becomes replan {lunches: 3, dietary: [vegetarian]}, which application code compiles into complete dish edits. If a physical fact is missing, the full proposal is stored with the question, and the user's confirmation applies it with zero model calls.
  • Hard limits and a budget ledger: at most 4 model calls, 6 tool calls and 3 changes per message, and 15 s per provider call. Each call's worst-case cost is reserved in a durable ledger before dispatch.
  • Durable recovery: each model decision is journaled before any tool runs, so a crash or retry resumes the recorded decisions instead of asking the model again.
  • Stack: Next.js 16, React 19, TypeScript, SQLite, vitest; browser Web Speech API for voice.

Challenges we ran into

  • The model invented a food-safety fact. Asked to switch to vegetarian, Nova 2 Lite repeatedly claimed "the chicken was kept separate" although the user never said so, even with an explicit rule against it. Prompting didn't fix it. Making such facts user-owned did: the app refuses the claim visibly, previews the change read-only, asks, and applies the user's confirmation without the model.
  • Low-level tools gave unpredictable results. With only add/remove/scale tools, the same request produced a different set of edits on each run. A single semantic replan operation compiled by code made outcomes predictable.
  • Cost control without token counting. Nova 2 Lite doesn't support CountTokens, so a guaranteed spending cap meant reserving the model's full context window per call.
  • No access to Alexa+ itself. The gated Alexa+ developer tools weren't available to participants, so we built our own web simulator and an MCP checker.

Accomplishments that we're proud of

  • The live Bedrock run on the final code passed end to end. Nova invented the fact, the app refused it, previewed the change through real MCP, asked, and the confirmation applied exactly one change with zero model calls, correlated with provider request ids.
  • All eight live scenarios have a passing record. Total live spend was 47 model calls and $0.396 counted against a $1.50 cap.
  • 514 automated tests, a 12/12 MCP end-to-end checker, and a 30/30 mock-provider checker including real crash-and-restart recovery.
  • Real microphone dictation works in desktop Chrome, with a review-before-Send step so a misheard word can never change the plan on its own.

What we learned

  • Treat physical facts as the user's, not the model's. Validation in application code, not prompt wording, is what made the model safe to use.
  • Design coarse, semantic tools for agents instead of CRUD-style tools.
  • Idempotency matters end to end. Request ids, receipts, journaling before dispatch and an "outcome unknown" state kept retries from duplicating changes.
  • Label modes clearly. Every reply says whether it came from scripted demo rules, a mock provider or live Bedrock.

What's next

  • Connect the MCP server to Alexa+ when the developer tools become available, for hands-free cooking.
  • Kitchen timers, equipment constraints (burners, oven) and multiple cooks.
  • A larger recipe set and broader live testing of natural-language requests.

Built With

Share this project:

Updates

Submission history