Inspiration
Your agent has a memory. It remembers what you said. It has no memory of what worked. So it makes the same mistake every week — it queries a table where amounts are stored in cents, gets the total 100× too big, you correct it, and next session it does the exact same thing. The fix never sticks, because "the fix" was never stored as knowledge — it just scrolled off. We wanted the missing half: memory of what works, that sticks.
What it does
Mulligan watches an agent succeed and fail and writes down what worked as a one-sentence lesson card — a testable hypothesis. It doesn't believe the card until it proves out: cards earn trust by working (win-rate, not vibes), and get retired to the graveyard when the world changes and they stop working. Learn, trust, forget — the full loop. It wires into any work-agent loop at three points: retrieve lessons before acting, record the outcome, learn from a failure (~10 lines).
How we built it
A deliberately boring agent works an 80-task text-to-SQL stream against a database with hidden quirks (cents, soft-deletes, epoch dates, status codes), graded by a deterministic verifier — execute the SQL, compare result sets, no LLM judge, so the score is objective. Four memory systems race on the identical stream: no-memory, Mem0, a ReasoningBank reimplementation, and Mulligan. At task 56 the world drifts — a rule that was true goes stale — and we watch who adapts. Everything runs on Qwen (qwen-plus) via Alibaba Cloud DashScope; every number is from a seeded, reproducible run.
Accomplishments we're proud of
Same 80 tasks, four systems, an objective verifier, four seeds. Mulligan learns at test time — in the headline run its rolling accuracy climbs 30% → 70% while no-memory stays flat. On raw accuracy it's a tie with Mem0 (45.8% vs 47.5%, reported undisguised) — but Mulligan gets there on 40% of the tokens, follows standing preferences twice as reliably (100% vs 50%), and it's the only system that recovers from drift: it detects its win-rate collapsing, retires the dead rule, finds the new source of truth, and a reborn successor card earns trust and takes over — recovery to 100% while the baselines stay flat at 0%.
Challenges we ran into
We caught a baseline-contamination bug: Mem0 silently routed its LLM calls to a different provider when a stray API key was in the environment, so early Mem0 numbers were secretly extracted by the wrong model. We found it, fixed it, and regenerated every baseline single-model. And one quirk (refund status codes) still fails: even when the environment reveals the correct code, the reflector thrashes among plausible variants and never commits — a clean lesson that observability is necessary but not sufficient.
What we learned
What the environment lets you disambiguate bounds what a test-time learner can learn. And that self-graded memory adds nothing: Google's ReasoningBank grades its own lessons with no ground truth, and it lands at no-memory level. Real outcomes plus trust is the whole difference.
What's next for Mulligan
A Mem0-backed card store (persisting cards through Mem0's storage — a clean side-by-side integration), more task domains beyond SQL, and a sharper reflector that commits to one hypothesis instead of thrashing.
Log in or sign up for Devpost to join the conversation.