The problem, in one sentence

"It's probably fine. Checkout retries failed jobs three times anyway."

In the reproducible checkout scenario, that was true in July. On 23 August the retry count was cut from 3 to 1 after a duplicate-charge incident. Nobody told the person wearing the Bee, because nobody tells anybody: the commit was reviewed, the config was deployed, and the change never reached the mental model of the engineer who is, right now, deciding not to investigate an alert.

Every team has instrumentation for the systems. None has instrumentation for what the people believe about them. That gap is where a class of incident lives: not a bug, not a misconfiguration, but somebody reasoning correctly from a premise that quietly stopped holding.

I see this in my audit work: what everyone believes the code does and what it actually does can quietly differ. Mental Model Drift checks the difference while the decision is still being made.

Try it

https://mental-model-drift.onrender.com provides an interactive instance using the non-private fixtures. Open a card's View evidence for the source, the locator and the timestamp; the Conversation review tab for the 98 utterances it decided not to speak about; the Agent check tab to hand it a sentence of your own and get back the verdict an agent would get. The public instance keeps the entrant's Bee account private; its search and Facts writes stay within the fixture proxy. It sleeps when idle, so the first load takes about a minute to wake.

What it does

Mental Model Drift hears a claim through Bee on a wearable or Apple Watch, decides deterministically whether it is a claim at all, and settles it against the system the registry names as authoritative.

  1. Hears the claim through Bee's realtime new-utterance stream, as spoken, while the decision is still being made.
  2. Decides it is a claim — deterministically, per clause. Questions, opinions, hypotheses, plans, directives and beliefs the speaker has already marked as past are dropped before any source is read.
  3. Verifies it: AWS AppConfig for deployed configuration and feature state, Sentry for the running release, the checked-in schema for structural facts.
  4. Explains it. This is the part that matters. It reconstructs the commit that moved the value, and searches the wearer's own Bee history for every earlier time they said the same thing. In the reproducible scenario: five conversations since 14 July, four of them while three was still the right answer. The output is not "you are wrong", it is "you were right, and then the software moved and nobody told you".
  5. Corrects it: writes the verified value into Bee memory as a confirmed fact, where the wearer's own assistant reads it next, and prepares a documentation pull request against the file that still says the old number.

And most of the time it does nothing at all. Across 115 utterances in the checked-in fixture corpus, 17 contain a checkable claim about five properties, three of which disagree with the reference configuration. The other 98 produce nothing. A tool that reads your conversations earns its place by staying quiet, so the dashboard has a Conversation review tab that shows the whole survey rather than asking to be trusted.

Four ways Bee is used, not one

A project that uses one Bee capability is a notification relay. This needs four, and each one is load-bearing:

Bee capability why the product cannot work without it
CAPTURE realtime stream, new-utterance the claim as spoken, while the decision is still being made
RECALL search --neural over past conversations one wrong sentence is a slip; the same one across six weeks is a mental model
RECONCILE changed --cursor realtime delivery is documented at-most-once, so it cannot be the record
CORRECT facts create / update the correction has to land where the wearer's assistant will read it next

RECALL is the one that makes the product exist. Without it the best this could say is "you are wrong". With it, it can say "you have said this in five conversations since July, and four of them were before the value changed", which is a completely different sentence and the only one worth interrupting someone for.

RECONCILE is the one that is easy to get wrong. Bee documents its stream as at-most-once, so I built the obvious thing: treat every disconnect as a gap and reconcile immediately. It recovered nothing, and passed every test I had. The conversation you miss is the one that happens while you are disconnected, and at the moment you disconnect it has not been recorded yet. Reconciliation has to run on reconnect. The recovery tests disconnect a stream, add a conversation with no subscriber attached, and require the cursor to recover it.

Bee integration and private data

Mental Model Drift has used data recorded and processed through an Apple Watch running Bee software, accessed through an authenticated bee proxy.

No private Bee conversation data is included in the public repository. Deterministic, non-private fixtures let judges reproduce the workflow without access to the entrant's personal recordings.

Three surfaces, one engine

Bee's own integration story is three doors, and the same question is worth asking through all three because the audiences arrive at different moments.

surface for when
the dashboard the person after the fact, with the timeline and the evidence
an MCP server an agent that speaks MCP mid-task, before it writes the patch
mmd + an Agent Skill an agent with a shell mid-task, no transport, no session

The second and third are the Assumption Firewall, and they are the part I would build next even if the hackathon ended tomorrow. A coding agent is handed human context constantly and has no way to tell a fact from a memory. Told "the worker retries three times, so a slow consumer isn't the problem", it writes a confident patch on a premise that stopped holding three weeks ago, and then defends it, because it reasoned correctly from what it was given.

check_assumption("The checkout worker retries three times, so a slow consumer is not the problem.")

  DRIFTED -- Checkout retry attempts
    stated 3, actually 1
    changed 2026-08-23 -- Reduce checkout retries to 1 after duplicate-charge incident
    severity HIGH
    restated in 6 earlier conversation(s), 2 of them after the change
    Act on the actual value, and tell the human what changed and when rather than
    silently correcting them.
    evidence: AWS_APPCONFIG OK appconfig://ecommerce/production/checkout-worker$.retry.max_attempts

The last line is the product. Knowing the value is not enough: an agent that silently substitutes it leaves the person still believing the old one, ready to repeat it in an hour, next to somebody else, where nothing is checking. Most of the Agent Skill is about how to say it.

The firewall lives in the engine rather than in any one surface, so a sentence gets the same verdict from the same registry through the same gate whichever door it arrives at. Exit codes carry it: 0 supported, 1 drifted, 2 inconclusive. Three, not two, because a connector that could not be read must never be indistinguishable from a person being wrong.

Why this is not an LLM wrapper

A model is never asked whether a statement is true. It is asked one question — which registry property is this sentence about? — and the answer passes through a deterministic gate before any source is read. Truth comes from AppConfig, Sentry and the repository. packages/drift-spec contains no I/O and no model at all.

Propose, then gate. Two independent proposers feed one grounding gate that requires the subject alias, a property lexeme and the value literal to actually appear in the spoken words, in a clause that asserts rather than asks. A proposer that invents a number cannot get past it:

"The checkout worker retries a bunch of times."   proposed max_attempts = 3
  rejected: the asserted value does not appear literally in the utterance

Four verdicts, never two. SUPPORTED / DRIFTED / INCONCLUSIVE / UNSUPPORTED_TYPE. There are 34 adapter tests, and every failure mode in them — timeout, 403, malformed document, absent property, sources that disagree — produces INCONCLUSIVE.

No competency score. The system never aggregates how often a person was wrong, and there is no per-person metric anywhere in the store. A tool that could produce one would be used to produce one.

Measured, not asserted

pnpm eval --verdicts runs 204 labelled utterances: 51 supported claims, 51 drifted claims, 51 technical-but-unverifiable statements, 51 non-claims.

grammar proposer, no model grammar + Bedrock target
candidate precision 100.0% 100.0% > 95%
recall 90.2% 93.1%
false-positive rate 0.0% 0.0%
subject / property mapping 100.0% 100.0% > 95%
value extraction 100.0% 100.0% > 98%
verdict accuracy 100.0% 100.0%
latency 2.9 ms / utterance 2.2 s / utterance

The second column is a real model on Bedrock, run over the whole corpus on 18 September. It raises recall and does not touch precision, because both proposers feed the same deterministic grounding gate and the gate is what holds precision. Recall is deliberately imperfect: the seven remaining misses are the gate declining on purpose, a negated clause or a sentence with no lexeme for the property, and each false positive avoided is a person not being told they are wrong about something they never said. pnpm eval --errors prints them.

Building that corpus found six real extraction defects in an afternoon, including one that inverted a claim: "the DLQ on the checkout worker is inactive" read on as a polarity word inside a prepositional phrase. None would have been found by more unit tests written by the person who wrote the code.

The part I am most glad I did

Integration tests run against tools/bee-sim, a test proxy implementing the documented /v1 wire protocol. When building client tooling, the danger is having a test harness share a misconception with the client: a test staying green while both sides share the same misconception.

That happened, and the test proxy and the client were both written from the documentation's examples, so they agreed on everything the examples get wrong. The examples show a stream frame with a speaker and no type, no utterance id and no time, and conversations with their words in a flat list. I concluded bee proxy was a subset of the CLI and built a hybrid client that shelled out for reads the proxy "did not have". All of it was wrong, and all of it had passing tests.

What Bee actually does came from @beeai/cli's source, which is MIT and one gh repo clone away, and from published live frames. A frame names its type on the SSE event: line and in its own type field, and carries the utterance's id and spoken_at, but no speaker. A conversation record has a numeric id and no uuid, keeps its words under transcriptions[] (a realtime pass, then a final one with new ids), gives every time in epoch milliseconds and every speaker as "Unknown". GET /v1/changes returns bare ids with its cursor at the top level; it is bee changed --json that moves it under meta. The proxy forwards every /v1 path upstream and has no route list at all. Against an account, the example-shaped client would have read every transcript as empty, never advanced its cursor, and never matched a live sentence to its conversation.

So the test proxy now answers in the API's shapes, and tests/conformance/bee-wire.test.ts holds it there: it copies parseSSEBuffer verbatim out of @beeai/cli, runs the literal wire bytes through Bee's own parser, and pins each field of a frame, a conversation read and the changefeed. A live sentence is joined to its conversation through the utterance id, the one thing the frame and the record share.

pnpm doctor checks the configured connection: it runs the application client and prints a row per capability, including whether the last realtime frame arrived named and carrying its utterance id, and whether a transcript read returns any utterances.

How it is built

TypeScript throughout, one pnpm workspace, 244 tests, tsc --noEmit clean.

packages/drift-spec/   types · adjudication · severity · validation   [no I/O, no model, no Bee]
packages/bee/          the only file that talks to Bee · event identity · fingerprints
packages/engine/       registry · speech acts · grounding · polarity · extraction · adapters
                       store · recurrence · pipeline · reconcile · capture · firewall
apps/server/           HTTP API, SSE, dashboard host
apps/dashboard/        drift cards, evidence, mental-model timeline, conversation review, agent checks
apps/mcp/              the Assumption Firewall over MCP
tools/cli/             `mmd`, the command the Agent Skill drives
tools/doctor/          the Bee preflight
tools/bee-sim/         wire-conformant test proxy for protocol conformance and failure injection
tools/eval/            the golden corpus and the metrics harness
skills/                the Agent Skill
infrastructure/        CDK: 35 resources
tests/conformance/     protocol wire bytes, through Bee's own SSE parser

AWS

service used for where
AppConfig + AppConfigData authoritative deployed configuration and feature state; hosted version history reconstructs when a value changed packages/engine/src/adapters/appconfig.ts
Bedrock (Claude via @anthropic-ai/bedrock-sdk, any other family via the API endpoint's chat completions) the second extraction proposer, asked only which registry property a sentence is about; over the corpus, recall 90.2% → 93.1% at 100% precision packages/engine/src/extract/bedrock.ts
DynamoDB single-table store: claims, evidence, drifts, cursor, dedupe markers with TTL packages/engine/src/store/dynamo-store.ts
CloudWatch the metrics that matter: how much was heard, how little was acted on, how often a card was dismissed infrastructure/lambda/index.ts
SQS, Lambda, API Gateway, Secrets Manager the deployed topology infrastructure/cdk/

cd infrastructure/cdk && npx cdk synth synthesizes 35 resources and bundles the handlers with esbuild from the same packages/ source the tests run against.

What I would build next

Team-scoped drift. Right now a belief is one person's. The interesting version is the moment two people on the same team hold different values for the same property, which Bee can see and nothing else can. That is the feature I would build first with more time.

Built With

Share this project:

Updates

Submission history