Dead Letter Club

Inspiration

Automation demos usually stop at the happy path, but production incidents begin when one step succeeds and the next one times out. That is the dangerous moment: replaying the whole job can duplicate invoices, emails, refunds, or shipments. We built Dead Letter Club to show how an agentic system can recover from that uncertainty without guessing, replaying blindly, or trusting a single success response.

What it does

Dead Letter Club consumes a dead-letter event, reconstructs observed state from CRM, Billing, Notifications, and trace evidence, and then chooses the smallest registered repair. Deterministic policy code approves or blocks the plan. The executor persists intent before each side effect, uses stable idempotency keys, reconciles ambiguous outcomes, and resumes from a durable graph cursor. An independent verifier reads every source system and emits a proof bundle only when the business invariant is satisfied.

The signature demo is real, not mocked: the private Billing service commits the invoice in Firestore, drops the HTTP connection, and terminates its actual Cloud Run process before the orchestrator can save a receipt. Cloud Run starts a fresh instance, and on resume the orchestrator discovers the existing invoice and continues without creating a duplicate.

How we built it

• Gemini 3.7 Flash on Vertex AI for evidence interpretation, minimal strategy selection, and independent verification
• Three private Google ADK semantic services with a Firestore-persisted execution cursor
• Pub/Sub dead-letter topic as the intake and retry boundary
• Firestore for source-system state, event deduplication, plans, intents, receipts, approvals, and verification reports
• Five scale-to-zero Cloud Run services: web, business systems, Investigator, Planner, and Verifier
• Cloud Trace and OpenTelemetry correlation across agent nodes and tools
• Next.js, TypeScript, Tailwind CSS, and Polar for the operator product and subscriptions

Challenges we ran into

The hardest design problem was separating “the call failed” from “the side effect did not happen.” We model timeouts as unknown, persist intent before mutation, and make every retry begin with a lookup by stable idempotency key. We also kept semantic reasoning away from authorization: the planner can propose only registered actions and cannot authorize itself.

Accomplishments that we're proud of

• Crash recovery with exactly one invoice and one payment email
• Independent source-system verification instead of trusting success responses
• Full-replay rejection whenever a non-repeatable receipt exists
• Evidence-backed unsafe escalation with zero mutations
• A resettable cloud demo with persistent failure scenarios

What we learned

Exactly-once is not a transport property. It is a business protocol built from idempotency, durable intent, reconciliation, and verification. Agentic systems become safer when language models interpret evidence while deterministic systems keep mutation authority.

What's next for Dead Letter Club

Add signed strategy registries, connector-specific least-privilege identities, compensation plans, tenant-defined policy packs, and production integrations for Salesforce, Stripe, and SendGrid.

Built With

• cloudrun
• cloudtrace
• firestore
• googleadk
• next.js
• opentelemetry
• pub/sub
• tailwind
• typescript
• vertexai