VIGIL

A crew is told an address is tier 1. They are never told why. VIGIL reconciles who depends on power to stay alive across four broken registries, the tier travels, the reason stays home.

Comment

Inspiration

Somewhere on your street there is a house where electricity is not a convenience. It runs a ventilator, or an oxygen concentrator, or the fridge holding someone's insulin. When the grid goes down, the utility is supposed to know that house is there.

Usually, it doesn't.

We started reading, and the numbers were worse than we expected:

  • A multiday blackout during a heat wave more than doubles heat-related mortality - modelled across 2.8 million residents of Atlanta, Detroit and Phoenix (Stone et al., Environmental Science & Technology, 2023)
  • The 2003 Northeast blackout: +122% accidental deaths, and +44% mortality among New Yorkers aged 65-74 ("Lights Out", Epidemiology)
  • After Hurricane Maria, device failure and interrupted care accounted for roughly one third of the estimated 4,645 excess deaths (New England Journal of Medicine)
  • 3M+ Americans depend on electrically powered home medical equipment (PubMed 26360818)

And then the part that actually shaped the project. The register that is supposed to hold these people fails in three specific ways:

  1. It's fragmented. In the UK a household must join four separate registers to be counted. In the US every utility keeps its own, joined to nothing.
  2. It's empty. Over 80% of UK adults have never heard of the Priority Services Register. Of those who have, 21% decline - over half citing embarrassment (Ofgem consumer research).
  3. It's stale. People move, conditions change, nobody re-verifies.

There is one genuinely good thing that already exists: HHS emPOWER, which maps Medicare beneficiaries on electricity-dependent equipment to ZIP codes, refreshed monthly. We consume it rather than compete with it. But emPOWER gives you a map of where these people probably are. It refreshes monthly, and an outage is measured in hours.

A map is not a memory. That gap is VIGIL.

What it does

VIGIL is a living record of who cannot afford to lose power, and it runs on one rule:

The tier travels. The reason stays home.

A crew anywhere reads "this address is tier 1." They never learn why - and the bytes that say why were never copied out of that person's jurisdiction.

End to end, it:

  • Merges four disconnected registries into one record. The same household appears three times under three spellings; VIGIL collapses them, and when two records are genuinely ambiguous it keeps them apart and flags them, because a wrong merge makes a household disappear.
  • Works out how urgent each household is, inside the region where the medical data lives. What leaves that region is an integer and a SHA-256 digest - nothing else.
  • Finds the people nobody ever asked. An enrolment agent infers likely eligibility and asks - then remembers who said no, permanently, because being asked twice about something you were embarrassed to disclose is the harm.
  • Dispatches welfare checks exactly once during a compound hazard (a blackout during a heat wave routes differently from either alone).
  • Records every attempt to read a clinical reason - granted or refused. A log of only successes can't show you who tried.
  • Reconstructs what it knew at 2pm that day, not what is true now - the only fair question to ask of the people who had to act on it.
  • Exposes all of it as an HTTP API and through the CockroachDB Managed MCP Server, for the questions nobody built a screen for.

This is what we mean by agentic memory. Not a record of a conversation - a record of who needs help, what was known, who was asked, and what happened.

Try it: main.drsi2hrw3uwzy.amplifyapp.com - sign in as a dispatcher, press reveal on a household, and watch it refuse. Then sign in as the Atlanta caseworker and press the same button. The difference is the whole project.

Every person in the register is invented. No real individual's health information is used, referenced, or approximated.

How we built it

The architecture is one decision

Two requirements, genuinely opposed:

  1. The register must survive losing a region - if it lives only in the datacentre that went dark with the grid, it is worthless at the one moment it exists for.
  2. The clinical reason must not leave its jurisdiction - replicating "ventilator, 15h/day" to two more regions to satisfy (1) is exactly what the law forbids.

CockroachDB refuses to do both in one database, and it is right to: PLACEMENT RESTRICTED on a database that survives region failure asks for a quorum that cannot exist.

So we split the memory along the seam the privacy rule already runs along:

Survival goal Holds
vigil SURVIVE REGION FAILURE identity, tier, hazards, dispatch, audit - no diagnoses
vigil_local SURVIVE ZONE FAILURE + PLACEMENT RESTRICTED diagnosis, device, tolerance hours, raw text - pinned per jurisdiction

The cost, stated plainly: lose a region and that jurisdiction's tiers survive - crews keep being dispatched - but the clinical justification is unavailable until the region returns. That's the right trade. A crew acts on a tier; it was never going to act on a diagnosis.

CockroachDB, and what each feature is load-bearing for

  • Distributed Vector Indexing (C-SPANN) - reconciliation runs exact match, deterministic rules, and an ANN search over 1024-dim identity embeddings, all inside one SERIALIZABLE transaction on one snapshot. With a bolt-on vector store the similarity query and the "who is already linked" query see different states, and the system creates the duplicate it exists to prevent. This is the reason the project is on CockroachDB rather than Postgres plus something.
  • Multi-region topology - two databases with opposing survival goals, and REGIONAL BY ROW driven by a computed column (crdb_region is STORED AS (CASE jurisdiction ...)), so physical placement is a deterministic function of the legal fact. Application code cannot misplace a row, because it never supplies the region.
  • AS OF SYSTEM TIME - reconstructing what was known, not what was true.
  • Row-level TTL - a register's confidence decays by default unless something re-asserts it.
  • ccloud CLI - the continuity agent reads regions and survival goals from the control plane; the chaos drill drives real region disruption.
  • Managed MCP Server - the investigator's door, scoped by GRANT.

AWS

Amazon Bedrock (Titan Text Embeddings V2 for identity projections; Claude only for genuinely ambiguous merges - every call's tokens land in a cost ledger) | Amplify Hosting (the public console and API, Next.js SSR) | Lambda (reconciliation sweeps - registry exports are bursty and peak exactly when staleness hurts most) | ECS Fargate (the long-running responder fleet) | EventBridge (hazard bus) | SQS FIFO + DLQ (every dead letter is a household nobody was sent to) | S3 (immutable adjudication evidence - the writer scans every bundle for clinical vocabulary and refuses, because S3 sits outside all residency boundaries) | CloudWatch (alarms on inference spend, dead letters, and residency violations) | ECR.

The redaction boundary

Registry free-text carries the reason inline, because that's what real registry data looks like:

"Mrs A. Iyer, 4 Elm Rd - oxygen concentrator, 15h/day"

One code path goes from pinned to replicated, and it uses two independent mechanisms: an allowlist (the projection is built by naming fields that may cross, never by removing ones that may not), and a 98-term clinical vocabulary scan on the finished projection, which rejects it if anything survives.

The second isn't redundant. Data entry puts the diagnosis in the wrong field constantly - name: "IYER A - VENTILATOR DEPENDENT" - and an allowlist copies that across without hesitation, because name is on the list.

Challenges we ran into

1. Residency and region-survival are mutually exclusive per database. This is the whole architecture, and we discovered it by having PLACEMENT RESTRICTED rejected. Splitting into two databases was not the plan; it was the consequence.

2. Multi-region configuration converges asynchronously, and nothing says so at the point of use. ALTER DATABASE ... PLACEMENT RESTRICTED returns immediately. The replicas then took 30-40 minutes to actually move, at roughly one range every five minutes. During that window a residency check reads as a violation when it is really a work-in-progress. We ended up asserting the declared policy and the observed placement as separate checks - and spent a long time unable to distinguish a slow rebalance from a broken configuration.

3. We documented a diagnosis we believed for weeks, and then disproved it. Writes to priority_tier as a GLOBAL table hung indefinitely (>60s vs 43ms regional). We concluded GLOBAL tables couldn't do read-modify-write, wrote it up in a migration comment, and believed it. Building an executable demonstration for an Agent Skill disproved it in about four seconds. We deleted the skill and corrected the migration. The root cause is still not isolated; the fix (REGIONAL BY ROW) is right on independent merit, and we say so rather than claiming an explanation we don't have.

4. Two bugs invisible at small scale. At 40 households precision read 1.0000. At 240 it read 0.9582 - conflicting initials treated as ambiguous rather than distinct, and the vector path auto-merging across different addresses because the name dominates the string. No distance threshold fixes the second. Both fixed; precision recovered to 0.9912.

5. A precision ceiling we cannot move. The one remaining collapsed record is two different households whose redacted projections are byte-identical. The raw pinned record has a phone number that separates them - and the reconciler can't see it, because reconciliation is cross-jurisdictional by definition and therefore runs on the projection. The privacy boundary imposes a precision ceiling. Only relaxing residency moves it.

6. An MCP client authenticates as a SQL user, so authorization living in application code doesn't apply to it. For a system with five roles that's a real hole, and it isn't obvious until you think carefully. Our answer was a dedicated SQL user with SELECT on the reason-free database and no grant at all on the pinned one - enforced by database privileges rather than by code the MCP path never executes. That turned out stronger than our application-layer checks.

7. A check that was always red. npm run score asserted zero wrong merges while the README, four sections earlier, documented the ceiling above as unreachable. The two contradicted each other and the check was wrong. A gate that is permanently red says nothing on the day it turns red for a real reason. Now bounded at the documented ceiling, so a regression still fails loudly.

8. ccloud auth login is an interactive browser flow, which is fine for an operator and useless for a scheduled continuity check or CI. We fell back to the Cloud API with a service-account key. Related: the Cloud API returns us-east-1 while the SQL layer returns aws-us-east-1 - comparing them literally reports every region as missing, and ours passed for days on the SQL fallback then failed the moment the better data source came online.

9. App Runner refused to start our container, so hosting moved to Amplify. The old path is still in the repo because Fargate needs the same build, and we say plainly that it no longer works.

Accomplishments that we're proud of

Everything prints a number that something else can contradict. Each of these exits non-zero on failure, against a live 3-region cluster:

Command Asserts Result
npm run db:residency the privacy guarantee, read from the replica map 8/8
npm run db:verify isolation, 5 exactly-once constraints, vector index, TTL 10/10
npm run access the boundary, both directions, with the log 6/6
npm run dispatch exactly-once under 8 responders racing 129 tasks 5/5
npm run continuity three regions, survival goals, time-travel reach 5/5
npm run skills both Agent Skills, on a live cluster 6/6
npm run mcp the investigator cannot reach vigil_local 3/3
npm run score precision 0.9912 / recall 0.7226, both error modes bounded 3/3
npm test tier ladder, digest stability, redaction logic 22/22

Specifically:

  • Precision 0.9912 at 240 households, with 462 of 535 links decided without a model call at all. The model is the exception, not the pipeline - using one where arithmetic suffices is a design failure, not a feature.
  • Zero clinical vocabulary in 433 globally-replicated rows, zero non-voting replicas, 100% of sensitive ranges physically converged - read from CockroachDB's own replica map, not asserted in a README.
  • 129/129 welfare tasks completed, none claimed twice, none stranded, across 8 concurrent responders.
  • Two Agent Skills contributed back, both shipping an executable demonstration that fails loudly if CockroachDB ever changes.
  • The interface is built to GOV.UK/NHS.UK conventions, not SaaS ones - never colour alone (every tier carries a number, a label and a colour, because red/amber/green is invisible to roughly 1 in 12 men), contrast >=4.5:1 verified by script, always-visible focus rings, reduced motion respected.
  • We published the things that didn't work, including a hypothesis we disproved and a ceiling we can't move.

What we learned

Writing a skill is a verification step. We set out to contribute three Agent Skills. Building an executable demonstration for one of them disproved a belief we had held for weeks, documented in a migration comment, and treated as fact. It took four seconds. We deleted the skill. A skill that merely asserts behaviour can rot silently - or be wrong on arrival, as ours was. Both surviving skills ship a demonstration for exactly this reason.

Benchmark a deduplicator at the scale it will actually run at. Precision 1.0 at 40 households was not a result. It was an absence of evidence.

Precision and recall are not symmetric when the rows are people. A missed merge costs one wasted visit. A wrong merge collapses two households, removes somebody from the register, and looks complete - so nobody notices. We resolve ambiguity to uncertain, never to a merge, and we let recall be imperfect on purpose.

A permanently-failing check makes every passing check beside it look negotiable.

The strongest privacy control was the boring one. Residency pinning is defence in depth. The control that actually carries the guarantee is that clinical text never enters the replicated database - so even a misplaced replica would be a replica of rows containing no diagnosis.

What's next for VIGIL

  • Real registry integrations - emPOWER as a live prior, plus utility and health-plan feeds under data-sharing agreements.
  • Secrets Manager or IAM database auth. The Lambda's database password is currently in an environment variable, and the Amplify build writes DATABASE_URL into the deployment artifact. Both are documented limitations, and for real health data both would be blockers.
  • Finish the Fargate responder image - the cluster, task definition, IAM roles and log group are deployed; the container is a placeholder and the loop currently runs locally.
  • File the two Agent Skills against the cockroachdb-skills repo. They're written to the spec; we didn't want to open issues during a judging window.
  • An external audit of the residency proof, because a privacy guarantee nobody outside the team has tried to break is a claim, not a control.

Every person in this project is invented. The shape of the problem is drawn from published research; the people are not.

References

  • Stone et al., "How Blackouts during Heat Waves Amplify Mortality and Morbidity Risk" - Environmental Science & Technology (2023), PubMed 37219950
  • "Lights Out: Impact of the August 2003 Blackout on Mortality in New York" - Epidemiology
  • "Mortality in Puerto Rico after Hurricane Maria" - New England Journal of Medicine
  • "Who's at Risk When the Power Goes Out?" - PubMed 26360818
  • HHS emPOWER Program - consumed as a data source, not competed with
  • Ofgem Priority Services Register research on awareness and refusal rates
  • NYC DOHMH heat mortality reporting

Built With

Share this project:

Updates