Sleeper

No single xz commit looked wrong. Sleeper's multi-year memory in CockroachDB sees the whole arc — and holds the backdoored release 34 days before the world found it by luck.

Comment

Sleeper watches a critical open-source package the way no human reviewer can: continuously, for years, remembering every commit, email and maintainer change. When a release is about to ship it rolls the contributor's entire trajectory into one behavioural arc, searches its own memory with CockroachDB's distributed vector index, and if that arc matches the shape of a known takeover it holds the release — atomically, with an audit trail a distro packager can query in one statement. Replaying the real public xz-utils timeline, it holds at the 5.6.0 upload: the moment the backdoor actually shipped, and 34 days before the real world found it by luck — 5.6.0 went out 2024-02-24, the backdoor was disclosed 2024-03-29.

Inspiration

On 2024-03-29 a backdoor was found in xz-utils (CVE-2024-3094), a compression library running on essentially every Linux machine on Earth. It had been planted over roughly two and a half years by a contributor who built trust one innocuous commit at a time — real bug fixes, plausible mailing-list posts, patient helpfulness toward an exhausted sole maintainer, two sockpuppet accounts pressuring him to hand over control, and finally release-signing authority.

What stayed with us is that no single commit was suspicious, and no single code review could have caught it. The tell existed only in the shape of the arc. And it was found five weeks after the poisoned tarball shipped, because one engineer got curious about 500 ms of unexplained SSH login latency. That is not a process — that is luck, and luck does not scale to the next one.

That framing made the memory layer the entire product rather than a feature of it. If the signal only exists across years of accumulated context, then whatever holds that context is the detector. Everything else is plumbing around it.

What it does

Sleeper is a release gate for a critical open-source package.

  1. Every signal — commit, mailing-list post, maintainer change, release — is embedded with AWS Bedrock Titan Text Embeddings V2 and written into CockroachDB.
  2. When a release event arrives, the agent reads that actor's history back out of the cluster (never out of the seed file), bounded to what was knowable at that timestamp, and rolls it into a single behavioural arc summary with Claude on Bedrock.
  3. The arc is embedded and searched two ways: a prefix-scoped ANN query over this package's own memory, and an unscoped query against a playbook of known takeover and ordinary-contributor arc shapes.
  4. If the arc is both close to a takeover shape and clearly separated from the nearest ordinary contributor shape, Claude composes a hold rationale and a distro advisory, and Sleeper commits the hold as one ACID transaction: insert the hold, flip the package's trust state, queue the advisory, write the audit row.
  5. A maintainer or downstream packager then asks "why did you hold my release?" and gets the whole evidence trail back — the matched arc, the similarity, the threshold in force, the EXPLAIN verdict and the structural evidence.

Step 5 is safe to paste as written. That read path now runs over the Cloud Managed MCP Server and has been verified end to end against https://cockroachlabs.cloud/mcp — all four read tools, with explain_query returning the prefix spans proof computed by CockroachDB itself. The transcript is committed at build/DEMO.md §2c. Direct SQL remains the tested fallback, and every run prints which path served the evidence.

(This block previously warned against claiming MCP at all, because the integration was unwired. Two of the three CockroachDB tools were at risk of failing Stage One. That is resolved.)

One thing in that list has not actually executed, and we would rather say it here than let you find it. Steps 1 and 2 describe Bedrock in the present tense because that is what the code does — but Bedrock is blocked account-wide on our AWS account. Every InvokeModel/Converse call, for Amazon Titan and Amazon Nova as well as Anthropic, in us-east-1 and us-west-2, from the Bedrock console playground as much as the SDK, returns ValidationException: Error 002: Access to Bedrock models is not allowed for this account. The Anthropic use case form is submitted and on file, and GetFoundationModelAvailability reports AUTHORIZED/AVAILABLE, so this is an account entitlement issue with an open AWS support case — not a missing integration.

What that means for everything else in this submission: no number here came from Bedrock. Every embedding in the live cluster carries embedding_model = offline-fnv1a-1024, and the single release_hold row is from the preview lane and labels itself [TEMPLATED TEXT — NOT MODEL OUTPUT] in its own reason column. The demo video says this on camera rather than glossing it.

What is real and reproducible, all of it visible in the video: the CockroachDB Cloud cluster with its three vector indexes, EXPLAIN proving the ANN scan was prefix-scoped to one package, the attack ranked out of memory by vector similarity, the four-write atomic hold, the privilege split refusing gate_svc a DELETE with SQLSTATE 42501, and the agent loop deployed and answering on AWS Lambda (aws lambda invoke on /health returns {"ok":true}).

How we built it

CockroachDB is the engine, not the store. Three inline VECTOR INDEX declarations. events is scoped on (package_id, embedding vector_cosine_ops), so ANN search is pre-filtered to one package's history rather than scanning the cluster. takeover_playbook is scoped on (held_out, embedding_model, embedding vector_cosine_ops) — both exclusions live in the index prefix rather than in a WHERE clause, because a held-out arc consuming a top-k slot would starve the two-sided gate of the benign neighbour it needs to measure a margin against.

We do not assert prefix-scoping in prose. EXPLAIN runs on the live query, the test suite fails if prefix spans disappears, and — the part we care about most — the same proof now comes back from CockroachDB Cloud's Managed MCP Server: explain_query returns prefix spans: [/'xz-utils' - /'xz-utils'] computed server-side, over the same statement the agent runs, to a client that could not have fabricated it.

The atomic hold is why this is CockroachDB rather than a vector database bolted onto a relational one: the vector search that produces the decision and the transactional state change that acts on it are the same system, so a hold can never half-land. There is no window where a release is blocked with no advisory queued, or an advisory goes out for a hold that was rolled back. We test that by killing a transaction mid-write and asserting nothing survives.

The Managed MCP Server serves the audit surface — the reads a distro packager performs on a hold they did not create. src/mcp.ts drives select_query, explain_query, get_table_schema and show_statement, binding argument names to the schema the server advertises in tools/list rather than hardcoding them. That decision paid on first contact: the real argument is query, not sql, and database is required on three of the four tools. The write path never uses MCP — one statement per call cannot express a four-write transaction, and pretending otherwise would break the one invariant the project rests on.

AWS Bedrock does all inference — Titan Text Embeddings V2 (InvokeModel) for every event and arc vector, Claude (Converse) for the arc rollup, the hold rationale and the advisory. Lambda hosts the agent loop, webhook-shaped: one event arrives, is embedded, written, and assessed against everything already in memory. ccloud provisions the cluster and a privilege split along the line that actually exists in the code — setup versus runtime: sleeper_admin owns DDL and the destructive setup paths, gate_svc runs the agent and holds no DELETE on any table, so the running agent cannot erase an event, a hold, an advisory or an audit row. Verified by execution rather than asserted: gate_svc attempting DELETE FROM events is refused with SQLSTATE 42501.

The stack is deliberately small — TypeScript on Node 22, pg, the Bedrock SDK, and Node's own http module for the demo. No framework, so everything a judge reads is the actual mechanism.

Challenges we ran into

Proving the retrieval was scoped, not just believing it. "Vector search on CockroachDB" is easy to claim and hard to verify. Getting EXPLAIN output that names the vector index and shows a bounded prefix span, then wiring that assertion into CI and putting it on camera, took longer than writing the query — and it is the single most load-bearing thing in the demo.

Not letting the benchmark be circular. The obvious version of this project trains on xz and then detects xz, which proves nothing. We split the corpus: the real xz timeline is the ground-truth hero replay and contributes to no reported number, while accuracy is measured only on held-out synthetic arcs that are excluded from every retrieval the agent runs. Thresholds are fitted by leave-one-out on the playbook split alone, never on the evaluation set, and the fitted file is gitignored so a threshold from a different embedding model cannot silently move the gate. We also run a lexical baseline over the same queries, because a similarity number means nothing without knowing what keyword matching alone would have scored.

Designing a gate that does not cry wolf. A bare "similarity ≥ X" rule flags every new, prolific contributor — which describes most good first-time maintainers, and would make the tool worse than useless in a real project. The decision became two-sided: hold only when the arc is close to a takeover shape and meaningfully closer to it than to the nearest ordinary-contributor shape. Several benign arcs in the playbook are written to superficially resemble takeovers so that second test has teeth.

Keeping the rules out of the decision. We compute structural signals — tenure, how fast trust escalated, what share of commits touch build machinery, which accounts pushed for a handover without contributing code. It was tempting to let them vote. We kept them strictly evidentiary, cited in the rationale but excluded from the decision, because mixing a hand-tuned rule engine into it would have turned the benchmark into a measurement of our rules rather than of the memory.

Accomplishments that we're proud of

  • The agent picks the suspect itself. With no actor configured, replaying the real xz timeline ranks candidates out of memory and assesses jia-tan — and the two runners-up are dennis-ens and jigar-kumar, the actual sockpuppet accounts that pressured the maintainer into the handover. Nobody told it where to look.
  • The prefix spans proof is produced by CockroachDB, not by us. It is in the demo, in npm run explain, in the test suite, and — via the Managed MCP Server's own explain_query — returned by the server to a client that could not have faked it.
  • The atomic hold survives an explicit mid-transaction kill test with no partial state, and commitUnhold gives it an exit that appends a resolution rather than deleting the record.
  • The structural evidence the agent extracts from memory is, independently, a correct reconstruction of the real attack: 862 days of tenure, privilege escalation 188 days after first activity, 43% of commits touching build machinery, two no-code accounts that argued for the handover and then went silent.
  • 588 tests. On a fresh clone with no database and no AWS account, npm test prints 522 passed | 66 skipped — and a DATABASE_URL that is set but unreachable skips those 66 with the reason printed (including the exact commands to start a cluster), because a stale credential should not look like broken code. Against a live CockroachDB node the full 588 run green.
  • The honesty scaffolding — held-out splits, leave-one-out calibration, a lexical baseline, and a bench script that refuses to compute accuracy when inference is stubbed, because a quality number derived from a hash function is a property of the hash.

What we learned

That "agentic memory" is a real architectural claim, not a synonym for a vector table. The moment we made the agent read its history back out of CockroachDB rather than from the file it ingested, the design got simpler and the point-in-time correctness problem became visible — a replay can otherwise leak hindsight into a past decision without anyone noticing.

We also learned how much of the work in a detection system is defending against your own optimism. Most of the engineering effort here went into making the result harder to believe: splitting the corpora, keeping calibration blind, adding a baseline, and building a mode that loudly refuses to produce numbers.

And the sharpest lesson came from the first live call to the Managed MCP Server, which disproved three things we believed. We had documented the audit path as "read-only at the protocol layer" on the assumption that a narrow cluster role yields a read-only tool list. It does not: tools/list is not role-filtered, insert_rows is advertised to an identity that cannot execute a single call, and no CockroachDB Cloud role grants MCP reads without also granting writes. The guard we had written to enforce that belief was worse than the belief — it refused any session advertising a write tool, which is every real session, so it would have silently disabled MCP in production while printing a security-shaped reason for the outage. What actually keeps that path read-only is that our client only ever builds SELECT, EXPLAIN and SHOW. That is discipline, not a boundary, and the difference is worth saying out loud. The enforced boundary is at the SQL layer, where gate_svc genuinely cannot DELETE.

The general version: binding to what a service advertises at runtime, rather than to what you assume it accepts, is the difference between an afternoon and a rewrite.

What's next for Sleeper

Multi-package monitoring with cross-package arc matching (an actor operating across several ecosystems is a much stronger signal than one in isolation). S3-backed build-vs-source tarball diffing, since in the real xz attack the malicious code lived only in the distributed tarball's build-to-host.m4 and never in git. Real webhook ingestion against live repositories. And calibration against a much larger corpus of reconstructed historical takeovers — the mechanic is only as good as the playbook it matches against.

Built With

Share this project:

Updates

Submission history