Chef Beside You

A multilingual AI cooking companion that sees, listens, and guides every step—from recipe discovery and shopping to the finished dish.

Comment

Inspiration

Cooking instructions assume everything is going according to plan. Real cooks need help when an onion is still too large, a pan may be smoking, or they are unsure which product to buy. Chef Beside You turns AI from a passive recipe reader into a calm, bilingual mentor that can observe, wait, and intervene.

What it does

Chef Beside You supports the whole journey from choosing a dish to shopping and cooking:

  • Searches 750+ dishes with multilingual intent understanding and semantic retrieval
  • Identifies a dish from a photo, then previews ingredients, method, and estimated time
  • Builds Japanese and English step-by-step cooking plans
  • Groups shopping items by supermarket section and checks product photos
  • Guides each cooking step on screen and by voice
  • Uses occasional changed still frames instead of continuously uploading video
  • Requires explicit confirmation before advancing when evidence is insufficient
  • Provides a clearly labeled deterministic demo for reliable judging

How we built it

The ASP.NET Core backend owns recipe state, timers, confirmations, and safety gates. TypeScript drives the responsive mobile UI, camera capture, text chat, browser speech, and OpenAI Realtime WebRTC sessions.

GPT-5.6 processes visual evidence with strict Structured Outputs and converts external recipe data into bilingual, observable steps. Hybrid retrieval combines LLM query interpretation with text embeddings. The Realtime API and OpenAI Agents SDK provide low-latency conversational guidance using ephemeral browser credentials.

Images and audio are not stored. The model never owns durable recipe state and cannot silently advance the workflow.

How Codex accelerated development

Codex served as the full-stack implementation partner throughout the core build session. It helped translate the product brief into a tested vertical slice, design the deterministic state machine, integrate official OpenAI SDKs, debug Realtime WebRTC, build bilingual accessibility, implement photo and shopping workflows, review security boundaries, add Docker support, and prepare the public repository and submission assets.

Challenges

The hardest part was balancing useful visual assistance with uncertainty and safety. A photo can show color, shape, and visible progress—but not temperature, freshness, allergens, or safe internal doneness. We addressed this with typed confidence, visible evidence, deterministic transition rules, and explicit confirmation.

Accomplishments

  • End-to-end recipe discovery, shopping, photo recognition, and cooking
  • Japanese/English UI and guidance
  • Privacy-conscious, event-driven image analysis
  • Server-verified progress rather than model-controlled state
  • 25 passing automated tests plus successful Release and TypeScript builds

What we learned

Strong agent experiences separate model reasoning from durable application state. Models interpret ambiguous evidence and communicate naturally; deterministic services enforce transitions, timers, and safety policy.

What's next

Persistent user recipes, dietary filters, store-specific aisle maps, and broader hands-busy learning workflows for crafts, repair, and accessibility.

Built With

Share this project:

Updates