Kitchen Agent

• 

  agent architecture

🍳 Kitchen Agent — A Voice-Native AI Cooking Companion

The Problem

Cooking is a hands-on activity, but most AI assistants are designed around answering questions rather than actually assisting someone through the cooking process.

When you're cooking, you don't want to repeatedly stop, pick up your phone, search for the next step, set a timer, or explain your situation from the beginning every time something changes.

We wanted to build an AI assistant that could stay with the cook throughout the entire process.

Our Solution

Kitchen Agent is a voice-native AI cooking agent that helps users plan, modify, and execute recipes through natural conversation.

Instead of behaving like a normal chatbot, Kitchen Agent maintains an active representation of the cooking session:

• 🧠 Planning — creates a structured cooking workflow.
• 📍 State tracking — remembers the current cooking step and completed steps.
• 🔄 Adaptation — handles ingredient changes and recipe modifications.
• ⏱️ Tools — performs deterministic actions such as creating, checking, and cancelling timers.
• 🎙️ Voice interaction — allows users to cook hands-free and naturally interrupt the assistant.

The goal is simple:

Don't make the cook adapt to the AI. Make the AI adapt to the cook.

How It Works

The system separates the cooking intelligence from the voice interface.

User Voice
    ↓
Real-Time Voice Agent
    ↓
Speech Recognition
    ↓
Kitchen Agent Core
    ↓
┌──────────┬──────────┬
│   PLAN   │   STATE  │   TOOLS  │ 
└──────────┴──────────┴
    ↓
Cooking Guidance
    ↓
Voice Response

The agent's core manages the recipe plan, current cooking state, completed steps, ingredient changes, and tool execution.

Deterministic actions such as timers are handled by Python rather than relying on the language model to maintain timing or state on its own.

For voice interaction, we integrated LiveKit, Silero VAD, Deepgram STT, OpenAI, and Rime TTS. This allows the assistant to support natural speech, interruptions, and background timer announcements.

What We Built

The project evolved through two stages.

Version 1 — Cooking Agent

We first built a terminal-based cooking agent to solve the core agent problem: how to make an LLM follow a cooking process instead of behaving like a normal chatbot.

We separated the system into planning, state, memory, tools, and reflection.

Version 2 — Voice-Native Agent

We then kept the cooking intelligence independent and added a real-time voice layer around it.

This separation means the same cooking brain can operate through different interfaces rather than coupling the reasoning system directly to the voice system.

Engineering Challenges

One of the biggest challenges was maintaining reliable cooking state.

LLMs can lose track of multi-step processes or change state unpredictably. We addressed this by keeping important state outside the LLM and using deterministic Python functions for state transitions and tools.

Another challenge was real-time voice interaction. Users may interrupt an assistant while it is speaking, so the voice layer needed to support interruption and conversation recovery.

Finally, cooking involves actions that happen over time. We implemented an independent timer system so timers continue running even while the conversation continues.

What We Learned

This project taught us that building an AI agent is not simply about giving an LLM a better prompt.

Reliable agents require state, tools, controlled actions, and clear separation between probabilistic reasoning and deterministic operations.

We also learned that voice changes the interaction model completely. A cooking assistant needs to be useful while the user's hands and attention are occupied.

What's Next

Future versions could add:

• 👁️ Vision-based ingredient detection
• 📷 Kitchen camera integration
• 🌎 Multi-language voice interaction
• 🧑‍🍳 Personalized cooking preferences
• 📚 A stronger recipe knowledge base

Kitchen Agent is our first step toward an AI assistant that can actively participate in real-world tasks rather than simply answer questions.

Built With

• agents
• ai
• deepgram
• language
• livekit
• natural
• openai
• processing
• python
• rime
• silero
• tts
• vad
• voice