Sous Chef | Your AI cooking assistant

Sous-Chef is an AI agent that eliminates kitchen friction. Using Gemini Live, it manages cooking tools, recipes, and tracks your progress via voice so you can focus on the food and not a messy screen.

Comment

Inspiration

This past month, I found myself in the kitchen baking almost every week. It was a rewarding process, but I quickly hit a recurring frustration: my hands were constantly covered in flour and dough, creating a messy cycle of washing and drying just to check a recipe on my phone or a piece of paper. I created Sous-Chef to solve this friction. It allows me to stay fully immersed in the joy of baking for the people I care about, ensuring every batch is perfect without the need to fight with a screen. With Sous-Chef, the focus remains on the food because the next step is always just a spoken question away.

What it does

Sous-Chef is a fully customizable AI cooking assistant that lets you define its name, personality, language, and even its voice to suit your kitchen vibe. Beyond simple conversation, it is equipped with a robust suite of culinary tools designed for a hands-free environment. It features precision utilities like integrated timers and recipe management tools that allow you to edit, save, or create recipes on the fly. Thanks to its multimodal capabilities, the assistant can "read" physical recipe cards through a camera tool or provide visual guidance on complex techniques. It even offers local intelligence, if you run out of an ingredient, Sous-Chef uses the location provided to find the nearest grocery stores. Most importantly, it maintains a contextual memory of your cooking session. You can track your progress simply by telling the assistant when you’ve added an item, and Sous-Chef will mark it off, even remembering your previous recipe if you happen to close the web-app without saving.

How I built it

The core of Sous-Chef is built on the Gemini Live API using the Google Gen AI SDK, which enables low-latency, natural dialogue that feels like talking to a human peer. To turn this dialogue into action, I implemented function calling for the timers, recipe management, and store discovery features. Data persistence for recipes is handled via the Local Storage API, while the Google Places API (New) powers the smart location-based searches. On the technical side, the frontend manages complex websocket connections to stream audio and video data in real-time, ensuring the assistant can see and hear the user without delay.

Challenges & Learning

While the documentation provided a strong foundation, the project presented unique technical hurdles that required creative problem-solving. I spent significant time perfecting the real-time audio visualizer -the dynamic "blob" at the center of the UI- to ensure it felt responsive and alive. Synchronizing the timer functionality with the live voice state was another complex task, requiring precise state management to keep the assistant aware of active countdowns during a conversation. Beyond the AI implementation, this project served as my gateway into cloud infrastructure; I learned how to containerize and deploy services using Cloud Run, ensuring the backend could scale and remain reliable.

This journey was a deep dive into the modern AI ecosystem, teaching me how to manage websockets for high-performance communication and architecting multimodal agents that maintain state over time. Pushing through these technical layers to build a tool that I will actually use in my own kitchen was an incredibly rewarding experience that has reshaped how I think about human-computer interaction.

What's next for Sous Chef | Your AI cooking assistant

The development of Sous-Chef is only beginning. Moving forward, I plan to implement enhanced safety features, such as emergency detection for smoke alarms and identifying dangerous kitchen habits via the camera feed. To make the agent more personalized, I aim to integrate Vertex AI Vector Search and a dedicated database to transform personal recipe collections into a searchable, organized library where every instruction feels as fluid as a poem.

To reach the next level of autonomy, I will explore reinforcement learning to adapt the agent's pace to individual cooking rhythms and a multi-agent workflow to handle specialized nutrition and planning tasks. Furthermore, I plan to implement voice biometrics and directional audio filtering. This will allow the agent to isolate the primary chef’s voice even in a noisy kitchen with background music or multiple people talking. By leveraging Gemini's multimodal reasoning alongside digital signal processing, Sous-Chef will remain focused only on the person it is assisting, ensuring accuracy and safety in busy household environments. Ultimately, Sous-Chef will continue to evolve into a smarter, more intuitive agent that makes the art of cooking faster, safer, and more enjoyable for everyone.

Built With

Share this project:

Updates