Molt

Inspiration

In March 2023 a Samsung engineer pasted proprietary semiconductor source code into ChatGPT to help debug it. The company banned employee use of external generative AI tools shortly after (Fortune, Mashable). What stuck with me was not the leak itself. It was that nobody could say afterwards exactly where that code had ended up, or prove it had been removed.

Now put that inside a dev consultancy running AI coding agents across a dozen client codebases, where every session feeds a shared memory that agents on any machine can read. That memory becomes a shadow copy of each client's source, secrets, and internal architecture. Worse, it blends them, because behavioural baselines and reusable procedures get distilled from work across several engagements at once.

Then one engagement ends with a contractual purge obligation, and somebody has to stand in front of a security reviewer who is paid to disbelieve them.

I went looking for the reason DELETE FROM memory WHERE client_id = ? is not an answer, and found two.

The first is semantic residue. Code pasted out of client A's repo into a session on client B's repo carries no path, no ticket number, no identifier of any kind. Exact match search cannot find it. Only vector similarity over the content itself can, which means erasure has to return probable matches and then commit to a decision about each one.

The second is blended artifacts. A behavioural baseline distilled from three clients has to lose one contribution and stay valid for the other two. Deleting it destroys memory the two remaining clients paid for. Keeping it is a breach. Neither option is acceptable, so the artifact has to be rewritten and the rewrite has to be checked.

Molt exists because those two problems make forgetting a database design problem rather than a delete statement.

What it does

Molt is shared memory for AI coding agents where a CockroachDB cluster is the only system of record, and the headline capability is provable forgetting.

The delivered path is capture, ledger, semantic recall, policy watcher, erasure certificate.

Capture hooks into five agent tools (Claude Code, Cursor, Codex, Gemini CLI, Copilot), plus an MCP proxy that records JSON RPC traffic in both directions and a decorator for in house Python agents. It strips secrets in process before anything hits the network, signs the batch, posts it over HTTPS, holds no database credential, and exits with status 0 in every branch including malformed input, because instrumentation that costs an engineer a session gets uninstalled.

The Collector verifies the signature and the request age, then appends events inside one serializable transaction where sequence numbers and hash chain digests are computed by the inserting statement itself. No process reads a digest and writes it back, because that round trip is exactly where a concurrent writer breaks the chain and a determined one forges it.

Recall sits on the agent's critical path rather than beside it. Before acting, an agent asks what happened the last time anyone in the fleet tried something similar, and the recorded outcomes change what it does next. Learned procedures carry a confidence value that rises on success and falls on failure, and procedures below the recall floor stop being served while staying in storage.

A policy watcher consumes the write stream as a changefeed and evaluates declarative rules whose actions escalate from a warning to halting the agent mid session.

Erasure runs in three phases under a fenced lease. Phase one is an explicit SQL sweep over sessions, events, attributed artifacts, lineage descendants, and embeddings. Phase two is vector search for residue, with a borderline band handed to a model that fails closed toward inclusion. Phase three decides each artifact individually: hard delete when no other client is bound to it, surgical rewrite when others are, and hard delete anyway if the rewrite fails validation.

Then it emits a signed certificate assembled by reading the stored evidence rows back, not from what the process remembers. The certificate ships with SQL a third party can run against the live cluster to reconfirm its central claim.

Every stored row lives in one of six memory tiers, and a tier exists because its rows have a different mutability contract and lean on a different database capability. The episodic ledger is append only and no role holds UPDATE. Attribution is an immutable version history, so "when did you first hold this" has an answer. Provenance is insert and delete only. Erasure evidence is write once, apart from the lease that governs who may write it. Working memory is disposable and swept by the cluster itself.

How I built it

CockroachDB is doing real work here, not holding rows.

Serializable isolation assigns ledger sequence numbers and chain digests inside the inserting statement, and makes attribution supersession and fencing generation assignment atomic instead of racy. Embeddings live in a VECTOR(1024) column behind a distributed vector index, and because the index I got reports the L2 operator class, I unit normalise every vector at write time. On unit vectors L2 ordering and cosine ordering coincide, so thresholds stay expressed in cosine space while the index does the work. Recursive queries walk the lineage graph with a cycle guard. Row level TTL makes the working tier genuinely disposable. Changefeeds feed the watcher. Four database roles enforce append only tiers and least privilege at the database, and migration 013 uses referential actions so deleting a row of audit evidence is refused by the cluster rather than avoided by discipline.

Fifteen SQL migrations in two generations, applied by a runner that records a digest per file and refuses to run when an applied migration has been edited.

AWS supplies the rest. Lambda hosts the ingest function and the console behind HTTPS function endpoints, so there is no idle server in the capture path. Fargate hosts the two components that hold long connections. KMS holds the asymmetric signing key, and verification retrieves the public key and checks locally, so verifying a certificate survives losing permission to call KMS. S3 stores certificates with versioning and Object Lock. Parameter Store standard tier holds every secret, chosen over the per secret store because I hold several secrets and that charge would have been the dominant avoidable cost. CloudFront terminates HTTPS on its own generated hostname. No load balancer, no NAT gateway, no interface endpoints anywhere.

Model access goes through an EmbeddingProvider or TextProvider interface, never a vendor SDK. Bedrock is the documented default. The selector refuses at startup any embedding implementation whose reported width is not 1024, which is what keeps the fixed column width honest.

The correctness work is where most of the effort went. Forty properties are specified in the design and each one is a Hypothesis test with a generator behind it. They assert things worth asserting: that a dry run and a residue sweep leave a byte identical digest across every memory content table, that no current attribution for an erased client survives a run, that a surgical rewrite leaves the other clients' bindings exactly as they were, that a certificate canonicalises to identical bytes under key and array shuffling, that altering any single byte of a signed payload is detected, and that recall never crosses a tenancy boundary.

Two gates run before any test. A strict type check with a type ignore allowlist that fails in both directions, so a stale entry is an error too. And a hygiene gate over eight pattern classes that refuses to let any tracked file carry a personal name, a date, a timestamp, a version history entry, or the name of a third party project studied while building.

Challenges I ran into

The garbage collection horizon was 4500 seconds. My original plan for certificate counts was a historical read at the run boundaries, comparing the cluster before and after. I measured the horizon rather than assuming a default, and 4500 seconds is far shorter than the lifetime of a certificate somebody might audit months later. So the primary count mechanism became derivation from the ledger and the stored dispositions, which depends on nothing that expires, and the historical read was demoted to opportunistic corroboration attempted only when both boundaries still fall inside the horizon. This was the single measurement with the largest design consequence.

Attribution supersession could not be one statement. Closing the current version is an UPDATE and writing its successor is an INSERT, and the cluster refuses to combine them, reporting that multiple mutations of the same table are not supported unless they all use INSERT. So supersession became two ordered statements in one transaction. That created a second problem: statement one writes a successor identifier for a row statement two has not inserted yet, so a self referencing foreign key would be violated between them. The cluster checks foreign keys per statement with no deferred checking available, so no arrangement of the constraint survives the ordering. I dropped the constraint and let the transaction carry the guarantee, with a partial unique index over unsuperseded versions keeping the ordering honest. The same reasoning applies to the lease and to the spawning event on a subagent session, which has to be inserted before the child session that references it.

On demand backup does not exist in the cloud control plane. I interrogated it rather than trusting a docs page, and it offers listing and configuration only. So the primary pre erasure backup path became a BACKUP INTO statement against an operator owned bucket, with a reference to the most recent managed backup as the fallback. The record distinguishes the two, marked taken or referenced, so a certificate never implies a backup Molt caused when it did not.

Inference quota was zero on the delivered account and not adjustable. A new account restriction outside my control, discovered after Bedrock was already the documented default. Rather than rewrite three components, I put the provider interface in front of them and moved the selection into configuration, then added the startup width gate so swapping providers cannot silently break the schema. The failure turned into the reason the abstraction is load bearing instead of decorative.

HTTPS on a load balancer was a dead end. No certificate can be issued for a load balancer's own generated hostname, so HTTPS there needs a custom domain, and no domain was in scope. CloudFront in front of a function endpoint terminates HTTPS on its own certificate with no hourly charge.

Failing closed is easy to say and awkward to implement. No lease means abort before any mutation. A failed backup means abort before any mutation. An unavailable adjudication means include the candidate. A rewrite that fails validation means hard delete the artifact. The last one took the most care, because a model that answers but answers badly is more dangerous than one that times out. The validator checks that the replacement is non empty after stripping, that the erased client's slug and display name and content markers are gone, that the length sits inside a ratio band so a degenerate one line answer is rejected, and that a retained client's marker is still present when the original had one. Anything failing is treated as unavailability.

Corrections to applied migrations cannot be edits. The runner records a digest per file, so editing an applied migration makes the next run refuse rather than report a clean no op. Every correction is a new numbered file, which is annoying once and correct forever.

Accomplishments that I'm proud of

The one I would show first is the fencing test. Ten real worker processes contend for one client's erasure lease. Exactly one wins and the rest are refused with a message naming the current owner and generation, so a loser learns who won rather than merely that it lost. I then kill the winner with no release and no final renewal, confirm a second worker is still refused while the expiry is in the future, let it pass, watch the takeover land with an incremented generation, and finally revive the dead worker and let it attempt a disposition write with the generation it still believes it holds. The database refuses that write with a stale generation error naming both generations, and no disposition row is persisted. Expiry is judged against the cluster's clock inside the transaction, never a worker's local clock, so a machine with a skewed clock cannot talk itself into a takeover.

Surgical redaction that actually preserves what it should. The rewrite is validated before it is accepted, the row keeps its identity and its revision history, the other clients' bindings come out untouched, and the erased client's binding is closed as history rather than left as a hole. The pre redaction body is copied into no table anywhere. What survives is the two digests and a structural summary of what changed, which is enough for a reviewer to see the shape of the edit without the removed content being retained in the name of proving it was removed.

Governance that rests on constraints instead of on my good intentions. No role holds UPDATE on the ledger, so append only is a privilege fact. Referential actions refuse the deletion of erasure evidence. The writer's UPDATE grants are scoped to individual columns. Signed checkpoints cover every session in a window rather than only the sessions a certificate names, and because the signing key lives outside the cluster, a consistent rewrite by someone holding database administrator access is still detectable. When a checkpoint does disagree, the verifier partitions the change: sessions whose differences are explained by recorded dispositions are reported as accounted for, and anything left over is the finding. A governed erasure gets explained rather than raised as an alarm.

Evidence written before, during, and after mutation. The run row exists with its before boundary before anything is deleted, every decision becomes a durable row as it is made, and the certificate is built by reading those rows back. A run that dies halfway leaves a readable account of exactly how far it got, which is the difference between an incident and a mystery.

Nothing important assumed. Four platform behaviours were probed live and recorded in a capability table the store reads at process start, and no component branches on a version string. Two of the four came back differently from what I assumed, and a third came back with a detail that changed the design: the index reports L2, not cosine. Each one changed the design rather than being worked around.

And a hygiene gate that fails my own build if a tracked file carries my name, a date, or the name of a project I read while building. It was an uncomfortable thing to point at myself and it has caught me more than once.

What I learned

Deletion and erasure are different problems. Deletion is about rows. Erasure is about content, including content copied into places nobody recorded, and derived content where the correct answer is a rewrite rather than a delete. Once you accept that, the search has to be semantic, the decisions have to be recorded, and the whole thing stops looking like a query and starts looking like a pipeline with evidence.

Probe the platform, do not reason about it. Every assumption I checked was cheap to check. Two of them were wrong in ways that would have been expensive to discover after the design had hardened around them, and a third came back carrying a detail that reshaped a schema decision. The habit that came out of it is that the answer lives in a capability table, not in my head and not in a version comparison.

Fail closed has to be the language the whole system is written in. If it is an error handler added at the end, half the paths quietly fail open, and those are precisely the paths an auditor finds. Writing it consistently means accepting the expensive outcome on purpose, over and over, in code that will mostly never run.

Property tests earn their cost on the invariants a demo cannot show. Ordering under concurrency, clamping at boundaries, candidate set disjointness, a prefix that has to be byte identical across a batch. All of that looks fine in a screen recording and is wrong in the data.

Cost is an architectural input, not a footnote. No load balancer, because its own hostname cannot hold a certificate and it bills by the hour anyway. No NAT gateway, because outbound only tasks in public subnets with no inbound rules do the same job. Standard tier parameters instead of a per secret store, because at this secret count that charge would have dominated everything else.

And the boring one that mattered most: writing the requirements and the design before the code. Fifty one numbered requirements and a design that specified every migration and every transaction boundary meant that when the platform surprised me, I could see exactly which requirements the surprise touched and change those, rather than discovering the consequence three components later.

What's next for Molt

Close the replay window properly. The request signature bounds it to the configured maximum request age rather than eliminating it, and a per request nonce table would close it at the cost of putting a write contended row in front of every capture. That tradeoff deserves a measurement rather than a preference, so the next step is to build both and benchmark the capture path under fleet load before choosing.

Harden the evidence posture for production. Object Lock in compliance mode instead of governance mode, with retention intervals set to the length of the obligation rather than to what makes a teardown script convenient.

Publish numbers for the one decision that changes erasure scope. The sensitivity grid already reports what each threshold pair would include, refer for adjudication, and recover against planted ground truth. I want that as a public benchmark with recall figures on a corpus anyone can regenerate, because a threshold with a measured recall curve behind it is a very different claim from a threshold somebody picked.

Push memory quality further than accumulation. Procedure confidence already moves with recorded outcomes and a floor keeps failing procedures out of recall. The next question is whether recall quality measurably improves over a fleet's lifetime, which needs the retrieval and outcome history turned into an evaluation rather than a display.

Take retention per jurisdiction seriously. Retention is enforced by the cluster today, and the interesting extension is regional placement, so a client under a data residency obligation has rows that physically live where the contract says they live and an erasure certificate that can say so.

Then breadth. More agent tools as their hook specifications land, and richer auditor tooling, because the reviewer sitting on the other side of that purge obligation is the person the whole system is built to satisfy, and the easier it is for them to interrogate the cluster themselves, the less any of this depends on being trusted.

Built With

• amazon-bedrock
• amazon-cloudfront
• amazon-cloudwatch
• amazon-web-services
• anthropic
• aws-cloudformation
• aws-fargate
• aws-kms
• aws-lambda
• aws-systems-manager
• boto3
• cockroachdb
• cryptography
• hypothesis
• jinja
• model-context-protocol
• mypy
• openapi
• psycopg
• pytest
• python
• sql
• starlette
• voyage-ai