Mental Model Drift: Bee hears it, production settles it

We monitor configuration drift, infrastructure drift and schema drift. Mental Model Drift monitors the one system nobody instruments: the engineer's understanding.

Comment

The problem, in one sentence somebody said

"It's probably fine. Checkout retries failed jobs three times anyway."

That was true in July. On 23 August the retry count was cut from 3 to 1 after a duplicate-charge incident. Nobody told the person wearing the Bee, because nobody tells anybody: the commit was reviewed, the config was deployed, and the change never reached the mental model of the engineer who is, right now, deciding not to investigate an alert.

Every team has instrumentation for the systems. None has instrumentation for what the people believe about them. That gap is where a class of incident lives: not a bug, not a misconfiguration, but somebody reasoning correctly from a premise that quietly stopped holding.

I see this in my audit work: what everyone believes the code does and what it actually does can quietly differ. Mental Model Drift checks the difference while the decision is still being made.

What it does

Mental Model Drift hears a claim through a Bee wearable, decides deterministically whether it is a claim at all, and settles it against the system the registry names as authoritative.

  1. Hears the claim through Bee's realtime new-utterance stream, as spoken, while the decision is still being made.
  2. Decides it is a claim — deterministically, per clause. Questions, opinions, hypotheses, plans, directives and beliefs the speaker has already marked as past are dropped before any source is read.
  3. Verifies it: AWS AppConfig for deployed configuration and feature state, Sentry for the running release, the checked-in schema for structural facts.
  4. Explains it. This is the part that matters. It reconstructs the commit that moved the value, and searches the wearer's own Bee history for every earlier time they said the same thing. In the demo: five conversations since 14 July, four of them while three was still the right answer. The output is not "you are wrong", it is "you were right, and then the software moved and nobody told you".
  5. Corrects it: writes the verified value into Bee memory as a confirmed fact, where the wearer's own assistant reads it next, and prepares a documentation pull request against the file that still says the old number.

And most of the time it does nothing at all. Across 115 utterances of recorded conversation, 17 contain a checkable claim and 3 disagree with production. The other 98 produce nothing, ever. A tool that reads your conversations earns its place by staying quiet, so the dashboard has a Heard tab that shows the whole survey rather than asking to be trusted.

Four ways Bee is used, not one

A project that uses one Bee capability is a notification relay. This needs four, and each one is load-bearing:

Bee capability why the product cannot work without it
CAPTURE realtime stream, new-utterance the claim as spoken, while the decision is still being made
RECALL search --neural over past conversations one wrong sentence is a slip; the same one across six weeks is a mental model
RECONCILE changed --cursor realtime delivery is documented at-most-once, so it cannot be the record
CORRECT facts create / update the correction has to land where the wearer's assistant will read it next

RECALL is the one that makes the product exist. Without it the best this could say is "you are wrong". With it, it can say "you have said this in five conversations since July, and four of them were before the value changed", which is a completely different sentence and the only one worth interrupting someone for.

RECONCILE is the one that is easy to get wrong. Bee documents its stream as at-most-once, so I built the obvious thing: treat every disconnect as a gap and reconcile immediately. It recovered nothing, and passed every test I had. The conversation you miss is the one that happens while you are disconnected, and at the moment you disconnect it has not been recorded yet. Reconciliation has to run on reconnect. The demo cuts the stream for real, holds a conversation with no subscriber attached, and lets the cursor recover it.

Three surfaces, one engine

Bee's own integration story is three doors, and the same question is worth asking through all three because the audiences arrive at different moments.

surface for when
the dashboard the person after the fact, with the timeline and the evidence
an MCP server an agent that speaks MCP mid-task, before it writes the patch
mmd + an Agent Skill an agent with a shell mid-task, no transport, no session

The second and third are the Assumption Firewall, and they are the part I would build next even if the hackathon ended tomorrow. A coding agent is handed human context constantly and has no way to tell a fact from a memory. Told "the worker retries three times, so a slow consumer isn't the problem", it writes a confident patch on a premise that stopped holding three weeks ago, and then defends it, because it reasoned correctly from what it was given.

check_assumption("The checkout worker retries three times, so a slow consumer is not the problem.")

  DRIFTED -- Checkout retry attempts
    stated 3, actually 1
    changed 2026-08-23 -- Reduce checkout retries to 1 after duplicate-charge incident
    severity HIGH
    restated in 6 earlier conversation(s), 2 of them after the change
    Act on the actual value, and tell the human what changed and when rather than
    silently correcting them.
    evidence: AWS_APPCONFIG OK appconfig://ecommerce/production/checkout-worker$.retry.max_attempts

The last line is the product. Knowing the value is not enough: an agent that silently substitutes it leaves the person still believing the old one, ready to repeat it in an hour, next to somebody else, where nothing is checking. Most of the Agent Skill is about how to say it.

The firewall lives in the engine rather than in any one surface, so a sentence gets the same verdict from the same registry through the same gate whichever door it arrives at. Exit codes carry it: 0 supported, 1 drifted, 2 inconclusive. Three, not two, because a connector that could not be read must never be indistinguishable from a person being wrong.

Technical details

A model is never asked whether a statement is true. It is asked one question — which registry property is this sentence about? — and the answer passes through a deterministic gate before any source is read. Truth comes from AppConfig, Sentry and the repository. packages/drift-spec contains no I/O and no model at all.

Propose, then gate. Two independent proposers feed one grounding gate that requires the subject alias, a property lexeme and the value literal to actually appear in the spoken words, in a clause that asserts rather than asks. A proposer that invents a number cannot get past it:

"The checkout worker retries a bunch of times."   proposed max_attempts = 3 rejected: the asserted value does not appear literally in the utterance

Four verdicts, never two. SUPPORTED / DRIFTED / INCONCLUSIVE / UNSUPPORTED_TYPE. There are 34 adapter tests, and every failure mode in them — timeout, 403, malformed document, absent property, sources that disagree — produces INCONCLUSIVE.

No competency score. The system never aggregates how often a person was wrong, and there is no per-person metric anywhere in the store. A tool that could produce one would be used to produce one.

Measurements

pnpm eval --verdicts runs 204 labelled utterances: 51 supported claims, 51 drifted claims, 51 technical-but-unverifiable statements, 51 non-claims.

grammar proposer, no model target
candidate precision 100.0% > 95%
recall 89.2%
false-positive rate 0.0%
subject / property mapping 100.0% > 95%
value extraction 100.0% > 98%
verdict accuracy 100.0%
latency 2.9 ms / utterance

The second column is a real model on Bedrock, run over the whole corpus on 18 September. It raises recall and does not touch precision, because both proposers feed the same deterministic grounding gate and the gate is what holds precision. Recall is deliberately imperfect: the seven remaining misses are the gate declining on purpose, a negated clause or a sentence with no lexeme for the property, and each false positive avoided is a person not being told they are wrong about something they never said. pnpm eval --errors prints them.

Building that corpus found six real extraction defects in an afternoon, including one that inverted a claim: "the DLQ on the checkout worker is inactive" read on as a polarity word inside a prepositional phrase. None would have been found by more unit tests written by the person who wrote the code.

The part I am most glad I did

Integration tests run against tools/bee-sim, a test proxy implementing the documented /v1 wire protocol. When building client tooling, the danger is having a test harness share a misconception with the client: a test staying green while both sides share the same misconception.

That happened twice. I concluded Bee's stream carries no event discriminator and wrote a classifier that guesses each frame's type from its payload. I concluded bee proxy was a subset of the CLI and built a hybrid client that shelled out for reads the proxy "did not have". Both conclusions were wrong, both were shipped, and both had passing tests — including one asserting a payload shape Bee never sends.

Twenty minutes with @beeai/cli's source, which is MIT and one gh repo clone away, disproved both. The stream's discriminator is the SSE event: name, which Bee's own parser requires. The proxy forwards every /v1 path upstream and has no route list at all.

So tests/conformance/bee-wire.test.ts now copies parseSSEBuffer verbatim out of @beeai/cli, opens a raw socket, and runs the literal wire bytes through Bee's own parser. Put the bug back and that parser returns zero events from the entire stream, which is exactly the danger: a client that drops the event name and a sender that never transmits one agree with each other and produce nothing at all in front of Bee's own parser.

pnpm doctor checks the configured connection: it runs the application client and prints a row per capability, including whether the last realtime frame arrived carrying its event name.

How it is built

TypeScript throughout, one pnpm workspace, 226 tests, tsc --noEmit clean.

packages/drift-spec/   types · adjudication · severity · validation   [no I/O, no model, no Bee]
packages/bee/          the only file that talks to Bee · event identity · fingerprints
packages/engine/       registry · speech acts · grounding · polarity · extraction · adapters
                       store · recurrence · pipeline · reconcile · capture · firewall
apps/server/           HTTP API, SSE, dashboard host
apps/dashboard/        drift cards, evidence, mental-model timeline, Heard tab, Agent tab
apps/mcp/              the Assumption Firewall over MCP
tools/cli/             `mmd`, the command the Agent Skill drives
tools/doctor/          the Bee preflight
tools/bee-sim/         the test suite's local Bee, wire-conformant, with failure injection
tools/eval/            the golden corpus and the metrics harness
skills/                the Agent Skill
infrastructure/        CDK: 35 resources
tests/conformance/     bee-sim's bytes, through Bee's own SSE parser

AWS

service used for where
AppConfig + AppConfigData authoritative deployed configuration and feature state; hosted version history reconstructs when a value changed packages/engine/src/adapters/appconfig.ts
Bedrock (Claude, via @anthropic-ai/bedrock-sdk) the second extraction proposer, asked only which registry property a sentence is about packages/engine/src/extract/bedrock.ts
DynamoDB single-table store: claims, evidence, drifts, cursor, dedupe markers with TTL packages/engine/src/store/dynamo-store.ts
CloudWatch the metrics that matter: how much was heard, how little was acted on, how often a card was dismissed infrastructure/lambda/index.ts
SQS, Lambda, API Gateway, Secrets Manager the deployed topology infrastructure/cdk/

cd infrastructure/cdk && npx cdk synth synthesizes 35 resources and bundles the handlers with esbuild from the same packages/ source the tests run against.

Bee integration and private data

Mental Model Drift has used data recorded and processed through an Apple Watch running Bee software, accessed through an authenticated bee proxy.

No private Bee conversation data is included in the public repository. Deterministic, non-private fixtures let judges reproduce the workflow without access to the entrant's personal recordings.

What I would build next

Team-scoped drift. Right now a belief is one person's. The interesting version is the moment two people on the same team hold different values for the same property, which Bee can see and nothing else can. That is the feature I would build first with more time.

Built With

Share this project:

Updates

Submission history