Kai

Kai is a voice-and-gesture kitchen agent on your fridge that tracks what’s inside, warns you before food goes bad, and walks you through cooking step by step.

Comment

Inspiration

Every week we threw out food we forgot we had: wilted spinach behind the milk, chicken that expired two days ago. And when we did cook, our hands were covered in flour or raw egg, so we couldn't scroll a recipe on a phone. We wanted something that lives where the food lives, on the fridge door, remembers what's inside, and talks you through dinner without you having to touch anything.

What it does

Kai is a small e-ink device that sits on your fridge, backed by a kitchen agent running on a Raspberry Pi.

  • Snap a receipt, stock the fridge. Take a photo of your grocery receipt and Kai reads it, turns each line into a food item with quantity, category, and a shelf-life estimate, and adds it to your inventory. Anything it can't read is flagged instead of guessed.
  • Knows what's about to go bad. Inventory is sorted by expiry. Items close to expiring are marked yellow and expired ones red, so you know what to use first.
  • Recommends what to cook. Ask "Kai, what should I make tonight?" and it ranks recipes by what you have, what's about to expire, what you've rated highly before, and your allergies and dislikes.
  • Cooks with you, hands-free. Kai walks you through a recipe one step at a time. Say "next," "go back," or "say that again," and run several named timers at once ("set a 10-minute timer for the pasta"). When you finish, it subtracts the ingredients from your inventory and asks how the dish turned out.
  • Works with hand gestures too. If the kitchen is loud or you'd rather not speak, four gestures control cooking: point up (previous), peace sign (next), open palm (repeat), fist (confirm).
  • A household web page. "My Fridge" lets anyone in the family edit the inventory, scan receipts, and set allergies from their phone. "Control Center" shows device status, conversation logs with timing, and lets you switch the model or Kai's voice.

How we built it

  • Device: ESP32-S3 with an e-ink display, ES8311 audio codec, speaker, microphone, and a push-to-talk button, written with PlatformIO/Arduino, in a 3D-printed stand. It talks only to our gateway over WebSocket (control, 16 kHz PCM audio streaming, live updates) and HTTP (uploads and audio clips).
  • Agent server: Python and FastAPI on a Raspberry Pi. The agent uses DeepSeek with streaming tool calls. It has about 25 tools for inventory, recipes, cooking steps, timers, preferences, screen pages, and voice, and it asks a yes/no confirmation before risky actions.
  • Voice: ElevenLabs streaming speech-to-text and text-to-speech, with switchable female and male voices.
  • Receipts: a vision LLM prompt that returns structured JSON, with unit conversions (oz → g, "12CT" → 12 pcs) and typical shelf life for each food.
  • Gestures: MediaPipe 21-point hand landmarks plus a rule-based classifier. No training data needed, and it runs on the Pi.
  • Teamwork: four people built four modules (inventory, agent and voice, gestures, firmware). We kept one shared interface spec (v1.7 by the end) and wrote mock services for each module, so everyone could build and test before the others were ready. Then everything was merged into a single server process with an automated test suite.

Challenges we ran into

  • Voice latency. Record, transcribe, think, and speak can easily feel slow. We stream at every stage: audio frames go up while you're still talking, the LLM streams tokens, and speech is generated and played sentence by sentence.
  • E-ink is slow. Full refreshes flash and take time, so we had to plan carefully which screens update and when, especially for timers and step changes.
  • LLMs like to guess. Early receipt scans invented items from blurry lines. We rewrote the prompt so the model must list lines it can't read instead of guessing.
  • Integrating four people's work. Field names, units, and message formats drifted between modules. A single spec with versioned changes and a rule of "announce, update the doc, then change the code" saved us.
  • Hardware surprises. The camera isn't wired on our current board, browsers only allow the mic and camera over localhost or HTTPS (we used SSH tunnels to the Pi), and MediaPipe's bundled OpenCV conflicts with a separate OpenCV install.

Accomplishments that we're proud of

  • A real end-to-end loop: receipt in, fridge stocked, "what should I cook?", step-by-step cooking with timers, ingredients deducted, dish rated, and better recommendations next time.
  • Cooking works completely hands-free, by voice or by gesture.
  • Four separately built modules came together into one system without a painful last-minute merge, thanks to the shared spec and mocks.
  • A gesture recognizer that works reliably with zero training data.

What we learned

  • Agree on interfaces first. Writing the spec and mocks before the real code let four people work in parallel.
  • Tool calling makes an LLM far more dependable than free-form chat. Giving the model narrow actions ("next_step," "set_timer") instead of asking it to describe what to do fixed most of our odd behavior.
  • On a voice device, perceived speed comes from streaming, not from a faster model.
  • Designing for e-ink and a single button forces you to cut everything that isn't essential.

What's next for Kai

  • Wire the camera on the device so receipt scanning and gestures work directly on the fridge.
  • Expiry reminders pushed to your phone, and an automatic shopping list from what runs out.
  • More gestures and better sign-language support for deaf and hard-of-hearing cooks.
  • Multiple user profiles in one household, with each person's tastes and allergies.
  • Smaller, cheaper hardware and a fully local model option for privacy.

Built With

Share this project:

Updates

Submission history