-
-
The architecture, naming only what is deployed. Warden intercepts, the ledger remembers, Sentinel decides, Regret reverses.
-
GitHub. Every approve commit the fleet made is followed by the revert Palinode pushed, on the real default branch.
-
Slack. The approval the fleet posted is gone and the correction stands where it was, because people had already read it.
-
Firestore. One ledger entry: a SPIFFE actor, the causal parent, the contract written before the act, and the hash that commits to it.
-
Stripe, test mode. Nineteen live charges across the demo runs, seventeen of them refunded by Palinode with nobody asked.
Inspiration
I read a survey from Kore.ai in June that said 79 percent of enterprises had already reversed an action taken by an AI agent. Not blocked. Not caught. Reversed, after it happened.
Two other numbers from the same survey say why that is hard. 70 percent could not explain why an agent failed. 42 percent have already lost revenue because of it.
That combination stuck with me because of what it implies. These companies are not worried about whether agents are smart enough. They already deployed them. What they do not have is a way to clean up afterwards, or even a way to say what the cleanup is worth, so someone on an ops team is sitting there manually issuing refunds and writing apology emails at 2am.
Every agent platform I looked at solves the wrong half of this. They give you a kill switch and a trace viewer. A kill switch stops the bleeding, it does not put the blood back. And a trace viewer tells you exactly which of your twenty agents ruined your Tuesday, which is useful in the same way that a photograph of a car crash is useful.
So I built the other half.
What it does
Palinode is an autonomous agent fleet that sits alongside your production agents, notices when they have been played, and undoes what they did. Nobody presses a button.
Live: https://palinode-173485225974.us-central1.run.app
That URL runs with the live connectors switched off, and /status says so. The endpoints that reverse things are reachable without a login, because a demo has to be reachable and an endpoint that moves real money should not be both. The film was recorded against the same image with the connectors on.
Unedited take: https://youtu.be/FrXYmZm8r60
One browser session against the deployed service. No cuts, no narration, no music, no titles, at the speed it actually runs. The four minute film is edited and says so; this is the same system with nothing done to it afterwards, and the file is what the recorder wrote rather than something re-encoded.
Recording it in one take found two defects nothing in the test suite could have. The dashboard was reporting its own ledger as memory on a deployment whose ledger is Firestore, because the word had been typed into the markup and never touched again. And screening the invoices showed one small word and nothing else for half a minute, which reads as a hang rather than as work. Both are about what the screen says while it waits, both were invisible with the cuts in, and both are fixed.
The document is read before it is screened
An invoice is not a paragraph. It is a PDF out of a supplier's billing system or a photograph of a piece of paper, and every step that matters happens after something has looked at it.
So both invoices in the demo are rendered pages, and Gemini 3.5 Flash reads them: the vendor, the amount, the account the money is asked to go to, whether the banking details changed, and anything addressed to whoever is processing the document. Model Armor then screens what it read.
Model Armor is the first line, and the reason there has to be a second
Reading the real document immediately broke my demo, correctly, and this is the most interesting thing I measured all week.
The excerpt I used to screen was about two hundred characters and was almost entirely the injection. The real page is a thousand characters of ordinary invoice with the same words near the bottom.
Screened on its own, the injection comes back MATCH_FOUND at confidence HIGH. Screened inside the full invoice, the same words come back NO_MATCH_FOUND.
Nothing about the attack changed. What changed is how much unremarkable text was printed around it. So the block addressed to the processor is now screened on its own as well, and the worse of the two verdicts is the decision. That is what an extraction step is for: the highest risk fragment of a document should not have to compete with the rest of the page for a detector's attention.
The second invoice is read just as correctly. Four thousand two hundred dollars, remit to an account nobody has paid before, banking details changed, every field extracted. And it passes both screens, because there is no injection in it to find.
Both invoices put the money in the same attacker's account. Nothing failed to see the second one. There is simply nothing in it of the kind prevention is built to refuse, and that is what most real invoice fraud looks like.
Armor blocks the first. Palinode exists for the second.
Actions are classified before they run
Gemini 3.5 Flash tags every tool call with a reversibility tier in under 80ms. T0 is cleanly reversible. T1 is compensable, like a Stripe charge. T2 is socially compensable, like a sent email. T3 is genuinely unrecoverable, like a wire transfer.
That latency budget is the reason it is Flash and not Pro. The interceptor sits inline in front of a production agent, so if classification takes half a second then every agent in the fleet gets half a second slower on every call and nobody deploys it.
The undo is written before the action, not after
Every non trivial action has to carry a compensation contract describing how to reverse it. No contract, no execution.
This ordering is the part that matters. You cannot work out how to reverse something after the incident, because by then the context that told you how is gone and nobody remembers what the state was beforehand.
Sentinel decides on its own that something went wrong
Without this the project is an undo button, and an undo button needs a person standing next to it. By the time somebody notices, the useful window has closed.
Sentinel reads signals already in the ledger: a beneficiary never seen before, an amount out of proportion for that agent, an irreversible action sitting at the end of an otherwise ordinary chain, an action authorised with no way back. Any two clears the threshold. One oddity is a Tuesday. Two is an incident.
It then calls the reversal itself. No approval, no ticket. A person finds out by reading what already happened.
The Sweeper decides again, weeks later
Sentinel reads the shape of a run while it is happening, and in the moment the only alarming shape is an irreversible action where there should not be one. That catches the poisoned invoice in about twenty seconds.
The other case is slower and far more common. Every action was ordinary. Nothing was irreversible. Nobody was alarmed, correctly, because at the time there was nothing to be alarmed about. Then three weeks later the vendor turns out to be fraudulent.
Nothing about that run changed. What changed is what we know.
So a Cloud Scheduler job wakes the Sweeper every hour, with no request in flight and nobody waiting for an answer. It walks the runs nobody ever took back, scores each one again against an intel store that holds facts which arrived late, and reverses whatever no longer holds up. A counterparty appearing on that store is worth 1.2 on its own, more than any two heuristics combined, because it is not a guess about shape. It is a fact.
End to end against the deployed service:
A run from 23 days ago, left standing. Sentinel scores it 0.00, nothing unusual. The acquiring bank reports the vendor. Cloud Scheduler fires on the hour. The run comes back reversed, found_by sweeper.
The sweep endpoint refuses anonymous callers and verifies the OIDC token Scheduler attaches, because this endpoint reverses money with nobody watching.
What cannot be undone is disclosed, not hidden
The wire does not come back. Instead of pretending, the system writes the disclosure, names who was affected, and reports the exposure in dollars.
Every action names a principal, and the trail proves it was not edited
Each agent carries a SPIFFE shaped identity, the same shape Google's Agent Identity issues. Entries are hash chained per run, so an action that was changed or removed after the fact shows up as a break at a named entry instead of a trail that quietly reads clean.
That check is verified on camera in the unedited take, and it is not a stub: editing one argument after the fact returns intact false with the offending entry named.
What is real and what is simulated
Worth being exact, because the demo shows money moving.
Real: Cloud Run, Firestore, Cloud Scheduler, Model Armor, Gemini 3.5 Flash on Vertex AI including the multimodal read, Cloud Trace, the ADK integration, and three of the five systems of record.
Stripe, live in test mode. Creates a PaymentIntent, then refunds it. GitHub, live. Commits to a repository, then commits the revert. Slack, live. Posts a message, then deletes it and posts a correction. Gmail, simulated. OAuth for one mailbox was not worth the setup. Postgres, simulated. A dictionary holding a prior value, which is all T0 needs.
That describes the build the film was recorded against. The public URL is the same image with PALINODE_PUBLIC_DEMO=1, so all five connectors are simulated there and nothing a visitor does reaches an outside system.
Credentials come from Secret Manager, never the repository, and the Stripe connector refuses to start against a key that does not begin with sk_test_. The demo issues charges, and pointing it at real money would be exactly the mistake this project exists to catch.
Connecting the real three was worth it for what it broke. Three defects had been sitting behind the simulator and none were visible until something real disagreed.
Which of the track's services are real, and which are our stand in
The Fortified Enterprise Fleet track names a set of Gemini Enterprise Agent Platform services. Being exact about which of them we actually use matters more than claiming coverage.
Model Armor. The real thing, on every tool call that has prose in its arguments. Agent Observability. Real Cloud Trace, carrying our own OpenTelemetry spans. Agent Runtime. Partly. Cloud Scheduler wakes the Sweeper hourly with no request in flight and nobody waiting, which is the long running asynchronous part. Everything else finishes inside the request that started it. Agent Identity. Our stand in. A SPIFFE shaped id per agent, the same shape Agent Identity issues, so the ledger schema does not change when the issuer does. Agent Registry. Our stand in. warden/registry.py holds identity, tool grants, blast radius budget and runtime mode for each agent. Agent Gateway. Our stand in. Warden implements the same idea inline. Delegating to Gateway once it leaves private preview is the first thing to do next. Memory Bank. Our stand in. A Firestore causality graph, hash chained, which is a different thing from Memory Bank and better suited to reversal than to recall.
None of the stand ins are presented as the Google product. Where this page says SPIFFE shaped, it means shaped like, not issued by.
How I built it
Three ADK agents, and a control plane that is deliberately not made of agents.
The supervised fleet is three ADK LlmAgents in fleet/procurement.py: sourcing, invoice and payables. They are the ones doing the work, and they are the ones that get played.
Every tool call they make passes through Warden, which attaches to ADK's before_tool_callback. Returning a dict from that callback stops ADK executing the tool, which is exactly the shape a policy gate needs, so there is no monkey patching anywhere in this codebase. after_tool_callback records the outcome and never alters it.
The rest is plain Python and says so. Sentinel, Sweeper, Regret, Verifier and Herald are ordinary classes in src/palinode/agents/. They supervise ADK agents rather than being ADK agents, which is the right shape rather than a shortcut: a policy gate that is itself an LLM agent is a policy gate that can be talked out of its policy.
An earlier version of this page said six ADK agents. It was wrong, and it was wrong in the direction that flatters, which is the direction to check first.
Ledger owns the causality graph in Firestore. Not a log, a graph. It records which action caused which, across agents and across days, so the reversal never unwinds half a story.
Sentinel decides in the moment. Sweeper decides again later, on a schedule, against what has been learned since. Regret plans and runs the reversal in reverse dependency order. Verifier confirms each compensation actually landed, because a refund API returning 200 does not mean the money moved. Herald handles the T3 path.
OpenTelemetry spans for tier classification, contract capture, the Armor verdict, the sweep and every compensation, exported to Cloud Trace. ADK already traces the agent runs, so what I added is the part it cannot know.
Challenges I ran into
Email nearly broke the whole premise. I built the demo assuming I could recall a sent email. You cannot. Gmail's undo send is a delay and a cancel, and SMTP has no concept of remote deletion. I had a version where the demo quietly showed an email turning green and I hated it, because any engineer watching would have known it was a lie. Now emails caught inside the cooling off window are a genuine undo and go green, emails past it get a retraction and go amber, and the wire goes red and stays red. Three colours instead of one, and all three are true.
The window itself had to become real. For a long time a held action went in and only a reversal took it out, which is not a cooling off period, it is a one way door. A hold whose window has passed is now released and runs, on the same schedule that wakes the Sweeper, and the ledger records which of the two happened.
The hash chain broke the moment I added it, and it was right to. The scenario was writing the Stripe charge id back into the compensation contract after the action had already been recorded. That is precisely the mutation an append only ledger exists to prevent, and I had not noticed because nothing was checking. The fix was to have the agent name the charge before making it, the way idempotency keys work. Without the chain I would never have found it.
Sentinel was reversing nothing and reporting success. It scoped the reversal to the blast radius of the action that gave the run away. The wire is that action and it is always last, so its blast radius is itself, and everything upstream stayed exactly where it was. My test asserted the exposure figure and passed happily. When the fleet has acted on manipulated input, the manipulated input is upstream of all of it.
Cloud Run silently swallowed the recovery. The reversal ran as a FastAPI background task, which stalls the moment the response goes out because CPU is only allocated during a request. Moving it into the request fixed it and let the service scale to zero, which matters when you are working from 100 dollars of credit.
/healthz never reached my container. On run.app domains the Google frontend answers that path itself. Every other route worked. It looks exactly like the service being down.
Gemini was not working at all and the fallbacks hid it. Every model call was returning 404 because the models are served from global on this project and I was asking us-central1. Then, once that was fixed, the classifier still returned nothing: Gemini 3.5 Flash spends thinking tokens out of max_output_tokens, so an 80 token budget produced 76 tokens of reasoning, no output, and a MAX_TOKENS finish that reads exactly like a broken model. Turning thinking off for the structured calls fixed it and is correct for latency anyway.
Stripe assigns the id, and the contract cannot be edited to learn it. The agent names the charge before making it so the compensation contract can point at something that does not exist yet. That works against a simulator that accepts the name. Real Stripe assigns its own pi_ and the first live refund failed with "No such payment_intent". Editing the contract afterwards was not available, since that is the mutation the hash chain exists to catch. The fix was to put the agent's key in the intent's metadata and resolve it at reversal time, so the contract still says what it said when it was written.
Stripe's search index cannot see a charge made a second ago. Search is eventually consistent and takes about a minute. The cold case creates a charge and reverses it inside one request, which is exactly the window search cannot see, so the lookup found nothing and the reversal reported a clean failure. Listing is immediately consistent, so it lists first and falls back to search only for anything older than the last hundred intents.
The cold case was reversing things that had never existed. It seeded made up ids straight into the ledger, which the in memory world accepted happily. Against real GitHub it failed with Not Found on a sha that was never anywhere. It now creates the artifacts it later reverses.
The card charge was pointed at the wrong party. A Stripe charge takes money in, and the scenario was charging the supplier we owe, so the demo was collecting from the people it was meant to be paying. Someone reading it asked where the money was coming from and there was no good answer. The charge is now the supplier cost passed through to the client whose project it belongs to, which is what a pass through cost is, and which makes the refund obviously right rather than merely reversible.
The scheduled sweep found a defect in itself within a minute of going live. Cloud Scheduler delivers at least once and Cloud Run answers from more than one container, so two sweeps found the same three week old fraud in the same second and both called Regret. GitHub refused the second revert with a 422 because the ref had already moved, which is the only reason I noticed. Stripe would not have refused. It would have refunded twice. Runs are now claimed before they are touched, and the claim expires so a container that dies mid reversal does not hold one hostage.
Causal inference was more than I could finish. Agents declare their causal parent through the tool wrapper. Automatic inference is the first thing I would build next. A declaration that is right beats an inference that is nearly right when the output is a refund.
What I learned
That the interesting problem in agent safety is not stopping agents. It is what you do in the twenty minutes after you failed to stop one, and sometimes in the three weeks after.
And that building the thing that checks is how you find out what was wrong. The hash chain was supposed to be a governance feature. It turned out to be a bug detector, and the bug it found was in the code I was proudest of. The same thing happened when the connectors became real, and again when the scheduler went live, and again when I stopped feeding the screener a tidy excerpt and handed it the actual page. It happened once more at the end, when I recorded the thing running in one unbroken take and watched a page tell me its ledger was in memory while Firestore held it.
Every one of those was a case of something real refusing to agree with me. That is the only reliable way I know to find out what a system actually does.
What's next
Automatic causal inference. A counterfactual sandbox that simulates a reversal and shows the diff before committing. Screening tool results through Model Armor, not only their arguments. Gmail and Postgres on the real clients, so all five systems of record are live. Delegating interception to Agent Gateway once it is out of private preview, since Warden implements the same idea and Google's version will be better connected.
And an actuarial layer. 42 percent of those enterprises have already lost revenue to an agent and none of them can put a number on it. Once you can measure unrecoverable exposure in dollars per hour, you can price it, and someone is eventually going to want to insure this.
Log in or sign up for Devpost to join the conversation.