-
-
The gate verdict: replaying 6 weeks (1,094 events) against v3 finds 3 duplicate purchase orders in 1.7ms — deploy blocked, exit 1.
-
Control run: the same gate replays v1 — the version that wrote the ledger — and returns 0 divergences. Proof it is not hardcoded red.
-
Two gates: idempotency at execution time, divergence at deploy time. The decision path is pure set arithmetic — the LLM blocks nothing.
Inspiration
Two incidents happened in our own system while building this. Both appeared only in the deployed service; the unit tests were green through both of them.
One — the same email went out twice. Pub/Sub delivered three ticks concurrently, two workers ran the same workflow step, and the ledger recorded mail.send#MSG-CA2473 twice. The idempotency check was ordered "look up, execute, record." Both workers looked, both saw no prior record, and both proceeded.
Two — a step disappeared silently. During a crash-resume test the workflow reported cursor 12/12 — complete — while only 11 side effects had actually occurred. A redelivery arrived inside the claim lease, so a legitimate retry was misread as a duplicate and skipped, and the cursor advanced anyway. No exception, no warning. The system reported success.
These two failures point in opposite directions. One did the same work twice; the other never did the work at all. What they share is that no human could have noticed either one. There is nowhere to ask whether a purchase order went out twice, and when a workflow says "done," nothing contradicts it. We produced both by running a six-week workflow for a single day. It is worth considering how often they occur in an agent that genuinely runs for weeks.
Google put the same question in the title of this hackathon's own workshop — "why does a resumed agent order two laptops?" An agent running in the background for weeks will eventually crash, eventually resume, and eventually have its code changed. At that moment, nobody is deciding whether an already-emitted side effect is about to be emitted again.
This is the friction Backstop removes: the question "if I ship this version, which of the last six weeks of work will it redo?" gets answered with an exit code instead of a guess. On our seed ledger the answer is three duplicate purchase orders, and the gate blocks the deploy.
Most submissions in this competition show what an agent accomplishes. We took the opposite side. We record what the agent did over the past six weeks, and before a new version ships we rewind those six weeks and compute what this version would have re-executed. The climax is not "the agent did it" — it is "the deploy was blocked."
What it does
Backstop has three parts.
1. The ledger. ADK's before_tool_callback and after_tool_callback intercept every tool call. Each call is written to Firestore with an idempotency key built from (tool name, canonicalised argument hash, run scope), alongside an OpenTelemetry span ID. This is the first gate, at execution time — a side effect with the same key does not go out twice.
2. Replay. The ledger's events are fed back, in order, to the new agent version. The tool executor is swapped for a no-op collector, so calls leaving the process number zero. Only the intents are gathered. Loading six weeks of ledger (1,094 events, 42 side effects), replaying it, and reaching a verdict takes 1.7ms.
3. The divergence gate. It takes the set difference between the side effects that actually went out and the intents collected during replay. There are three outcomes — DUPLICATE (about to redo something already done), MISSING (no longer does something it used to), MUTATED (same target, different value). A single DUPLICATE returns exit code 1 and the deploy is blocked.
In Backstop, the LLM blocks nothing. The gate is set arithmetic, a pure function, with unit tests attached. If divergence.py imports a network or model library, CI fails. Gemini 3.5 Flash is called only to write the one explanatory sentence at the bottom of a divergence card. The replay and decision path make zero model calls; generating the three explanations costs three calls (capped at five). Remove the model entirely and DEPLOY BLOCKED and exit code 1 are unchanged.
How we built it
The subject-agent is an ADK agent running a six-week vendor onboarding workflow. A Pub/Sub agent.tick message advances it one step, and state persists in Firestore. It is only the specimen under audit, so it never exceeds five tools.
The core is how the replay harness swaps the execution mode. Inject IntentCollector into ADK's tool execution path and the agent believes it is calling tools while nothing leaves the process. To enforce that, a test monkey-patches sockets — a single network call during replay turns it red. Blocking sockets alone was not enough, so the test also asserts the stubs' side-effect lists stay empty, because our stubs need no network to cause an effect.
Time is injected everywhere. No datetime.now() call exists anywhere in the code; everything goes through clock.now(), and an AST-based test enforces it across the repository. That is what let us generate six weeks of ledger with no real waiting, and because that ledger is committed, anyone can run make replay with no API key and see the same divergences.
Challenges we ran into
Six of them. The first two are fixed, and how they were fixed is most of what this product is. The remaining four are unsolved.
Fixing the two incidents above changed the design twice. The duplicate leaked because gate ① was ordered "look up, execute, record." The key is now claimed before the tool runs, and the Firestore document ID is the idempotency key, so uniqueness is enforced by a storage constraint rather than by application logic. That introduced the opposite bug — a crash redelivery landing inside the claim lease was misread as a duplicate, and the cursor advanced past work that never happened. So blocking now means two different things: blocked by a committed effect is work already done, so advance the cursor; blocked by an in-lease claim is work still unfinished, so hold it. The gate's DUPLICATE and MISSING categories are not invented taxonomy — they are the names of these two incidents.
A crash after claiming but before committing is indistinguishable. Dying just before the tool executes and dying just after it executes but before the record is written look identical in the ledger. The window is milliseconds, but it is real, and reclaiming in the second case sends the side effect twice. Today we only buy time with a 25-second lease. The real fix is a lookup API on the tool side or a two-phase commit, and we left it out because stubs cannot prove it works.
Changing a prompt shifts the idempotency key. When argument canonicalisation changes, a semantically identical call produces a different key. The demo's three divergences are exactly this — v3 changed vendor_id from "acme-corp" to "ACME Corp", purely cosmetically, and the gate calls it duplicate purchase orders. That is not a gate malfunction: the order really would go out a second time. The limitation is that the gate cannot separate "a harmless relabelling" from "genuinely a different vendor." We version the canonicalisation rules (CANON_VERSION) so a rule change is detectable, but the judgement is not automatic.
The six-week ledger is generated by simulation. It is not a real six-week operational log; it is compressed into existence with a FrozenClock. The event distribution and failure rates are values we chose, and it does not carry the messiness of real production traffic.
We did not integrate Memory Bank or Agent Registry. They are recommended Fleet-track components, but the learning cost did not fit the sprint. Memory is served by a Firestore implementation behind the same interface.
The Narrator fabricates. One of the three divergence explanations written by Gemini asserted the matter "has already been logged in an existing incident" — a fact that does not exist. So on screen this text is the lowest, dimmest line of the card, and make gate defaults to deterministic pre-written sentences. The verdict is identical either way. This is the one place in the product that is genuinely an LLM wrapper, which is exactly why it is isolated from the decision path.
What's next for Backstop
Managing the ledgers of several agents registered in Agent Registry behind one gate; today it handles a single agent's ledger. And joining Model Armor's blocked events into the ledger, so the gate can also compute "this version would have let 2 of the 12 past prompt-injection attempts through."
Log in or sign up for Devpost to join the conversation.