Inspiration
Households throw away food they already paid for, mostly because nobody is watching the pantry. We wanted an agent that does the watching and the deciding on its own, but that is honest about when it should not act.
What it does
Autonomous Pantry Agent runs a household pantry without being asked.
- It detects triggers: an item about to expire, or stock about to run out.
- It plans one dish for two people around that item, using what is really in stock.
- It reserves the ingredients, checks the plan against real inventory, and looks for gaps.
- If an ingredient is missing, a Resolver searches the web with Linkup and an LLM extracts a sourced substitute.
- If the plan is safe and complete, it approves itself and updates the inventory. If not, it stops and waits for a human (approve, reject, or give feedback).
- Every run keeps a decision trace, and any run can be undone.
In the demo, one click starts a three-day auto run: day by day, the agent plans, gates, approves and records a dish with zero further clicks.
How we built it
- MongoDB Atlas holds the inventory as an event-sourced store (idempotent events, invariants such as quantity >= 0 and reserved <= quantity), plus run history and caches. Atlas Vector Search powers the agent's memory of past decisions.
- A worker process drives each run through a state machine: triggered, planned, gated or pending confirmation, approved, recorded.
- Linkup provides sourced web answers for the Resolver; Claude extracts structured substitutes from them. Embeddings use text-embedding-3-small.
- A Streamlit UI shows live inventory, a run timeline, the decision trace and command status.
- Plans and Resolver answers are cached so the demo is repeatable.
Challenges we ran into
- Safe autonomy is harder than autonomy. Staples that are not tracked, items at zero stock, and substitutes whose names did not match the pantry each blocked runs in different ways, and each needed its own fix and tests.
- Units: when a substitute is given in a different unit (for example millilitres versus grams), the agent refuses to guess a conversion and blocks the run instead.
- Making an LLM-driven demo repeatable: we used temperature 0 and a plan cache so the same input gives the same plan.
- Keeping a three-person team's work in one codebase under time pressure.
Accomplishments that we're proud of
- A real three-day, zero-click run on real Atlas and a real LLM, rehearsed repeatedly.
- A gate that blocks rather than guesses: an insufficient item never gets auto-approved, and a missing one goes through the Resolver first.
- Over 200 automated tests covering the inventory invariants, triggers, gaps and the worker.
- A standalone Linkup demo script that shows the Resolver end to end.
What we learned
- The hard part of an agent is deciding when not to act, not generating the plan.
- Event sourcing makes undo and audit almost free.
- Deterministic caching turns a flaky live demo into a reliable one.
What's next for Autonomous Pantry Agent
- Multi-day planning instead of reacting day by day.
- Wiring guest allergies into the gate so risky dishes are blocked automatically.
- Resuming a paused run, and letting humans approve a plan that uses a substituted item.
- Receipts and barcode scanning to fill the pantry without typing.
