Sous Recall

Inspiration

Cooking with a recipe on your phone is messy: your hands are covered in flour or oil, and you keep wiping them just to scroll. Voice assistants help, but they forget who you are. If you're allergic to peanuts, you have to say it again every single time, and the one time you forget, the suggestion can be dangerous. We wanted a kitchen assistant that talks with you like a friend, remembers what you can't eat, and proves it checked.

What it does

Forkget-Me-Not is a voice sous-chef that never forgets your allergies.

1. Snap your fridge: a vision LLM suggests one simple dish from what it sees.
2. It asks first: on your first visit it asks, by voice, "Any allergies or foods you avoid?" Returning users go straight to cooking.
3. Cook hands-free: hold to talk and say "next", "repeat that", or "how much salt?" Replies are short and spoken aloud, and a full-screen cook mode shows one big step at a time.
4. Remembers and updates: mention a preference at any time ("I don't eat spicy food anymore") and it's saved. Newer facts supersede older ones instead of piling up, and allergies can only be removed if you explicitly say so.
5. Double-checks itself: every recipe and reply is evaluated against your preferences by an LLM judge, and the result is shown as a badge ("✅ Checked against 3 preferences").

How we built it

• Frontend: a single vanilla HTML/JS page with cook mode, an animated voice orb (idle / listening / thinking / speaking), and a live "What I remember about you" panel.
• Backend: Python + FastAPI.
• LLM: Claude for vision and conversation. Each voice turn is one structured JSON call that returns the spoken reply, the intent (next / repeat / answer), and any new preference to remember.
• ElevenLabs: Speech-to-Text (Scribe) for every voice command, Text-to-Speech for every reply.
• SurrealDB Agent Memory: preferences stored as typed facts (allergy / dislike / diet) with timestamps; new facts supersede old ones, so history is kept and memory persists across sessions.
• Arize AX: end-to-end tracing of every request (STT, LLM, TTS spans with latency), plus a preference-violation LLM-as-judge eval logged on every output.
• [如果做了] Daytona: the app runs in a Daytona sandbox and is served from its preview URL.

Pipeline: voice → STT → LLM (recipe + current step + current preferences) → preference check → TTS → browser.

Challenges we ran into

• Keeping spoken replies short enough to be useful in the middle of cooking.
• Deciding when a new preference should replace an old one, while making sure an allergy can never be silently overwritten.
• Making the safety check visible to the user instead of hiding it in logs.
• Building and integrating three sponsor platforms in about four hours.

Accomplishments that we're proud of

• A fully hands-free loop that actually works live: photo → recipe → voice-guided cooking.
• Memory that survives a page refresh and correctly handles changing tastes.
• A safety check that is visible on every single answer, not just a line in the system prompt.

What we learned

• Memory needs time and types: "what's true now" matters more than "everything ever said".
• Prompt instructions alone aren't a safety guarantee; an independent eval on every output is cheap and catches mistakes.
• Observability from minute one (Arize traces) made debugging the voice loop much faster.

What's next for Forkget-Me-Not

• Wake word instead of push-to-talk, plus voice-controlled timers.
• Household profiles: cook for several people and merge everyone's constraints.
• Allergen detection directly on the fridge photo.

Built With

• anthropic
• api
• arize
• audio
• ax
• claude
• css
• elevenlabs
• fastapi
• html
• javascript
• mediarecorder
• opentelemetry
• python
• surrealdb
• web