Inspiration
AI agents make work cheap to start but surprisingly expensive to recover.
When an agent session is interrupted, restarted, or handed to a fresh worker, useful investigation can disappear from the active context. The next worker may repeat the same analysis. More dangerously, an external action may already have succeeded before the worker crashed, and a naive retry can perform the action twice.
Recovery Taskmaster was built around one principle:
Do not retry uncertainty. Reconcile it.
Instead of treating model output as success, the system treats success as a verified state transition.
What it does
One request starts the complete recovery workflow:
Complete a recovery workflow for run_id judge-demo-01.
Do the work end to end and stop only at VERIFIED or an explicit blocked state.
Gemini 3.5 Flash, orchestrated through Google ADK, works through a small set of scoped tools to:
recover persisted evidence
→ establish current state
→ materialize one bounded recovery action
→ independently reread the resulting state
→ VERIFIED | BLOCKED | FAILED
The agent chooses the next scoped operation, but deterministic host code owns the safety-critical boundaries:
- evidence existence
- finding authorization
- legal state transitions
- path containment
- permitted mutation
ACTION_PENDINGpersistence- ambiguous-execution reconciliation
- final verification
This makes Recovery Taskmaster an execution workflow rather than a chatbot.
The failure case that matters
The most important path is not the happy path.
Recovery Taskmaster explicitly handles:
ACTION_PENDING persisted
→ side effect succeeds
→ worker crashes before recording success
→ fresh worker starts
→ reread external/current state before retry
→ effect already exists
→ do not execute again
→ independently verify
→ VERIFIED
A hostile recovery test confirms that the effect occurs only once.
A separate reconciliation harness applies the same boundary to the GitHub Contents API: it performs a real remote mutation, simulates losing local settlement, then requires a fresh process to inspect remote state before deciding whether retry is safe.
This does not claim universal exactly-once execution. It demonstrates a concrete reconcile-before-retry pattern for queryable side effects.
How I built it
The contest-specific Recovery Taskmaster layer uses:
- Gemini 3.5 Flash for agent reasoning and orchestration
- Google Agent Development Kit (ADK) for the agent/tool workflow
- Vertex AI for Gemini access
- Google Cloud Run for the hosted agent runtime
- Python for deterministic host logic
- SQLite for persisted recovery evidence/state
- GitHub Actions
- GitHub OIDC → Google Workload Identity Federation for keyless deployment
- GitHub Contents API for external reconciliation testing
The deployed proof path is:
GitHub OIDC
→ Google Workload Identity Federation
→ exact-source Cloud Run deployment
→ remote Gemini / ADK execution
→ terminal VERIFIED state
→ public sanitized receipt
What I learned
The biggest lesson was that agent reliability is not primarily about making the model more confident.
The difficult boundary is between:
"I called the tool"
and:
"I independently verified that reality now matches the intended result."
That distinction changes the architecture.
It requires persisting ambiguity before mutation, restricting the agent's authority surface, rereading state independently, and treating verification—not model prose—as the terminal success condition.
I also learned that human-style retry logic is dangerous for autonomous agents. When execution status is uncertain, immediately repeating the operation can increase damage. Reconciliation is often the safer next operation.
Challenges
The hardest part was proving the system beyond local code.
Recovery Taskmaster needed to demonstrate that:
- the exact source was actually deployed,
- Gemini through Google ADK actually orchestrated the remote workflow,
- the workflow reached a real
VERIFIEDterminal state, - deployment identity was bounded and reproducible,
- ambiguous execution did not silently become a duplicate action.
The final deployment therefore publishes a sanitized receipt containing the source commit, Cloud Run service and revision, model, workflow status, and whether the remote trace actually reached VERIFIED.
Pre-existing work disclosure
Recovery Taskmaster itself was created during the hackathon submission period.
It incorporates Operational Waste Recovery (OWR) as an explicitly disclosed pre-existing dependency. OWR supplies persistence, deterministic repeated-work detection, evidence review, and Recovery Capsule generation.
The hackathon-specific work is the new Gemini/ADK Taskmaster layer, including:
- four bounded agent tool contracts
- autonomous workflow orchestration
- host-owned execution state machine
ACTION_PENDINGambiguity boundary- reconcile-before-retry behavior
- independent verification
- Cloud Run deployment
- keyless Google authentication
- external reconciliation harness
- hostile failure tests
- live execution receipts
- hackathon-specific documentation and demo
Proof
Live Cloud Run service
https://recovery-taskmaster-dzoo5fey5q-ww.a.run.app
Source
https://github.com/leadingproblemsolver/operational-waste-recovery
Recovery Taskmaster
Verified deployment receipt
The hosted receipt proves the exact source was deployed to Google Cloud Run and that Gemini through Google ADK completed the bounded workflow to a captured VERIFIED state.
It does not claim customer adoption, realized financial savings, production scale, universal exactly-once semantics, or hackathon placement.
Log in or sign up for Devpost to join the conversation.