LetAISous

Got your hands dirty when cooking, and you need the next step to the recipe? LetAISous handles everything in the kitchen for you

Comment

Inspiration

Cooking is when you least want to touch a screen. hands are messy, timers overlap, and it's easy to lose your place in a recipe. We wanted a sous chef you can simply talk to, one that knows what's in your kitchen, keeps your place, and keeps track of every timer. Alexa+ on a kitchen display felt like its natural home.

What it does

Talk Sous is a voice-first AI sous chef, built as an Alexa+ kitchen-display prototype (a browser app with an Echo Show–style interface).

  • Pantry and recipe discovery: tell it what you have ("add 6 eggs and 500 g rice to my pantry") and ask what you can cook. It searches 13 curated recipes by your ingredients, suggests substitutions, and as a last resort creates a clearly labelled AI recipe that must pass food-safety checks.
  • Hands-free cooking: it reads one step at a time, remembers your place, answers questions like "is chicken safe at 150°F?", and handles interruptions.
  • Timers: multiple named timers that survive page reloads and restarts, with spoken and on-screen alerts.
  • Servings and pantry updates: scale the recipe, see what you're short on, and when you finish, choose to subtract the ingredients from your pantry (with undo).
  • Your units: amounts follow your country's measurements, metric or US.

How we built it

  • Frontend: a single web page using the browser's speech recognition and speech synthesis.
  • Backend: Python and FastAPI with SQLite. The app, not the AI, owns every session, recipe step, per-user pantry and timer. Timers are stored as absolute deadlines, so they stay correct across restarts.
  • AI: Amazon Bedrock (Nova Lite) through the Converse API with tool calling. Actions are exposed as 18 tools on a self-hosted MCP server over Streamable HTTP. Our chat path uses an in-memory MCP client against that same server, so one tool definition serves both the app and external MCP clients.
  • Reliability layer: plain phrases ("set an 8 minute timer for eggs", "next", "switch to US units") are handled by deterministic code, and guards replace anything the model claims to have done but didn't.
  • Food safety: curated safe-temperature data, plus validation of any AI-generated recipe.
  • Security: an access code with a signed cookie, per-user data isolation, rate limits, a strict content-security policy, a bearer key for the MCP endpoint, and a least-privilege IAM role.
  • Deployment: Amazon EC2 using an IAM instance role (no AWS keys on the server), Docker Compose, and Caddy as a reverse proxy with automatic HTTPS.
  • Testing: 363 backend tests using a scripted fake Bedrock, plus 65 browser-page checks.

Challenges we ran into

  • The model said it did things it hadn't. Nova Lite reported timers that didn't exist and recipe steps that weren't in any recipe. We moved state and common commands into code and added guards that replace unsupported claims.
  • Bedrock region and permissions. In Singapore we had to call the model through an inference profile, and our first least-privilege IAM policy had to be widened to cover it.
  • Voice needs HTTPS. Browser speech recognition only works over HTTPS, so we added a domain and Caddy's automatic certificates.
  • MCP inside one app. Calling our own MCP endpoint over HTTP deadlocked the server, so we switched to an in-memory client.
  • Getting the numbers right. Parsing "2 cups of rice" into grams, scaling servings, and making ingredient subtraction happen exactly once, even if you double-click, with undo.

Accomplishments that we're proud of

  • A demo that behaves the same way every time for the core flows, because the app owns the logic.
  • Timers and pantry that survive page reloads and server restarts.
  • Subtracting ingredients after cooking exactly once, with undo, even under concurrent requests.
  • A public deployment on AWS with per-user data isolation, rate limits and no AWS keys on the server.
  • Tests that we deliberately broke to confirm they catch regressions.
  • A written friction log of what slowed us down, with suggestions.

What we learned

  • Don't trust a small model with state. Let the model request actions and let the application decide, then test the ways the model fails.
  • MCP in practice: one tool definition for both an agent and external clients, Streamable HTTP, and the trade-offs of securing it without a full OAuth flow.
  • AWS details: Bedrock inference profiles, least-privilege IAM for them, and using an instance role's credentials from a container.
  • Safety belongs in code, not only in prompts: curated temperatures and validation held up where prompts alone did not.
  • Deploying a small stack: Docker Compose with Caddy for automatic HTTPS on EC2.

What's next for LetAISous

The immediate next step is expanding the per-user pantry system by integrating computer vision to automatically parse and log grocery store receipts. We also plan to implement multi-user household support so family members can share a synced pantry while maintaining individual cooking sessions. Ultimately, when the Alexa+ SDK exits preview, we aim to deploy this application natively onto physical kitchen display hardware.

Built With

Share this project:

Updates

Submission history