Quorum

An active-active incident control plane on Aurora DSQL that stays strongly consistent even when a whole region, including your observability region, goes dark

Comment

Inspiration

When an AWS region goes down, the tools a team uses to coordinate the response often live in that same region, so the outage takes the incident response down with it. Quorum is an incident command plane that survives the region it runs in: an event-sourced coordination system on Amazon Aurora DSQL, multi-region active-active, with a Next.js front end on Vercel. It is built for any team whose incident response cannot afford to share fate with the region that just failed.

Inspiration

About twenty years in infrastructure and SRE taught me one failure mode that keeps recurring: the status page, the chat bridge, and the runbook tooling are hosted in the region that just failed, so you end up running a region failure over a status page the region failure already knocked offline. Aurora DSQL is the first managed primitive I have used where a coordination system can be active-active across regions with strong consistency and no single point of failure, so I built the tool I always wanted during an incident.

What it does

Quorum is a live incident command plane. Responders open and acknowledge incidents, add notes, and resolve them, and every action is an immutable event appended to a log rather than an in-place update, so the timeline is its own audit trail. A Reliability surface proves the guarantees on the click, not from canned numbers: a cross-region write, a no-split-brain race between two writers on the same record, a fifty-write concurrency burst, and a failover drill that opens a real sev1 incident which you then coordinate from the surviving region. When both regions are down, it reports the true state instead of faking a write: committed data is safe in the witness log, and writes resume on recovery.

How I built it

The data model is event-sourced across four append-only tables, with the event UUID serving as both primary key and idempotency key. Concurrency uses optimistic concurrency control: no row locks, conflicts resolved at commit time, the losing transaction retries. That single property does two jobs. It is why the cluster recovers cleanly when a region drops, since there are no stranded locks to reconcile, and it is the same reason two responders cannot fork the incident record.

The stack is TypeScript end to end, Next.js App Router on Vercel, Kysely for queries instead of an ORM, and Aurora DSQL as the multi-region store: us-east-1 and us-east-2 as full regions, and us-west-2 as a log-only witness for commit quorum. The app authenticates to DSQL with IAM through Vercel OIDC federation, so there are no static credentials anywhere. Incident signals arrive through a CloudWatch alarm to EventBridge to an ingest Lambda that writes into DSQL, and a scheduled monitor Lambda revalidates failover and writes status through the same database, so the health panel reads from the store that survives the outage.

Most of the code was written by directing Claude Code. What kept it coherent across a large surface was an append-only architecture decision log: every decision numbered, committed separately from the code, never edited after the fact, with a fifty-test suite gating every merge.

Challenges I ran into

The honest one first: as deployed, the serving tier is single-region. The data plane survives a region loss with no data loss, but making the Vercel serving tier multi-region is a plan-gated step I scoped out rather than fake, and the demo is precise about that line. Getting the demo to tell the truth in the failure case took real care. The chaos toggle simulates a region endpoint going unreachable and exercises the application's failover without pretending it destroyed an AWS region, and the both-regions-down state reports data-safe rather than a successful write. A coordination tool that lies about its own state in the failure case is worse than no tool. Keeping provisioning open for judges while stopping abuse also took two passes on a rate limit tracked in DSQL itself.

What I learned

The strongest result was conceptual: in a lock-free, commit-time-conflict system, partition recovery and write-conflict safety are the same mechanism, not two separate features. I also learned how far agent-assisted development goes when it is governed. The agent is fast and good at the local task; what it does not have is memory of why something was decided or any feel for what a change costs three decisions later, and an append-only decision log is what supplies that.

What's next

Multi-region serving to match the multi-region data plane, a per-workspace retention sweep, and rolling the coordination model into adjacent civic and crisis-response problems where strong consistency on shared state is the right tool.

Everything in the live demo runs on the free tier. After all the load testing, it has only used a fraction of the 100K free-tier allowance. Full write-ups on the data model, the failover, and the build are in the series linked below.

Built With

Share this project:

Updates