-
-
Mental Model Drift catches an engineer reasoning from a value that has quietly changed in production.
-
A premise that was true in July and wrong in September, and nothing that checks what people believe.
-
The spoken claim, the verified value from AWS AppConfig, and the earlier conversations where it was said before.
-
The evidence: source, address and timestamp, and when the value changed against when it was repeated.
-
Most of what Bee hears produces nothing, and the review tab shows the whole survey instead of asking to be trusted.
-
A coding agent sends the same sentence over MCP or one command and gets the same verdict, told to say what changed.
-
Four Bee capabilities, each load-bearing: capture, recall, reconcile and correct.
-
From pnpm eval --verdicts: the deterministic gate holds precision, and the Bedrock proposer raises recall.
-
Bee on one side; AppConfig, Sentry and the repository as sources of truth; one engine behind three surfaces.
The problem, in one sentence
"It's probably fine. Checkout retries failed jobs three times anyway."
In the reproducible checkout scenario, that was true in July. On 23 August the retry count was cut from 3 to 1 after a duplicate-charge incident. Nobody told the person wearing the Bee, because nobody tells anybody: the commit was reviewed, the config was deployed, and the change never reached the mental model of the engineer who is, right now, deciding not to investigate an alert.
Every team has instrumentation for the systems. None has instrumentation for what the people believe about them. That gap is where a class of incident lives: not a bug, not a misconfiguration, but somebody reasoning correctly from a premise that quietly stopped holding.
I see this in my audit work: what everyone believes the code does and what it actually does can quietly differ. Mental Model Drift checks the difference while the decision is still being made.
Try it
https://mental-model-drift.onrender.com provides an interactive instance using the non-private fixtures. Open a card's View evidence for the source, the locator and the timestamp; the Conversation review tab for the 98 utterances it decided not to speak about; the Agent check tab to hand it a sentence of your own and get back the verdict an agent would get. The public instance keeps the entrant's Bee account private; its search and Facts writes stay within the fixture proxy. It sleeps when idle, so the first load takes about a minute to wake.
What it does
Mental Model Drift hears a claim through Bee on a wearable or Apple Watch, decides deterministically whether it is a claim at all, and settles it against the system the registry names as authoritative.
- Hears the claim through Bee's realtime
new-utterancestream, as spoken, while the decision is still being made. - Decides it is a claim — deterministically, per clause. Questions, opinions, hypotheses, plans, directives and beliefs the speaker has already marked as past are dropped before any source is read.
- Verifies it: AWS AppConfig for deployed configuration and feature state, Sentry for the running release, the checked-in schema for structural facts.
- Explains it. This is the part that matters. It reconstructs the commit that moved the value, and searches the wearer's own Bee history for every earlier time they said the same thing. In the reproducible scenario: five conversations since 14 July, four of them while three was still the right answer. The output is not "you are wrong", it is "you were right, and then the software moved and nobody told you".
- Corrects it: writes the verified value into Bee memory as a confirmed fact, where the wearer's own assistant reads it next, and prepares a documentation pull request against the file that still says the old number.
And most of the time it does nothing at all. Across 115 utterances in the checked-in fixture corpus, 17 contain a checkable claim about five properties, three of which disagree with the reference configuration. The other 98 produce nothing. A tool that reads your conversations earns its place by staying quiet, so the dashboard has a Conversation review tab that shows the whole survey rather than asking to be trusted.
Four ways Bee is used, not one
A project that uses one Bee capability is a notification relay. This needs four, and each one is load-bearing:
| Bee capability | why the product cannot work without it | |
|---|---|---|
| CAPTURE | realtime stream, new-utterance |
the claim as spoken, while the decision is still being made |
| RECALL | search --neural over past conversations |
one wrong sentence is a slip; the same one across six weeks is a mental model |
| RECONCILE | changed --cursor |
realtime delivery is documented at-most-once, so it cannot be the record |
| CORRECT | facts create / update |
the correction has to land where the wearer's assistant will read it next |
RECALL is the one that makes the product exist. Without it the best this could say is "you are wrong". With it, it can say "you have said this in five conversations since July, and four of them were before the value changed", which is a completely different sentence and the only one worth interrupting someone for.
RECONCILE is the one that is easy to get wrong. Bee documents its stream as at-most-once, so I built the obvious thing: treat every disconnect as a gap and reconcile immediately. It recovered nothing, and passed every test I had. The conversation you miss is the one that happens while you are disconnected, and at the moment you disconnect it has not been recorded yet. Reconciliation has to run on reconnect. The recovery tests disconnect a stream, add a conversation with no subscriber attached, and require the cursor to recover it.
Bee integration and private data
Mental Model Drift has used data recorded and processed through an Apple Watch running Bee
software, accessed through an authenticated bee proxy.
No private Bee conversation data is included in the public repository. Deterministic, non-private fixtures let judges reproduce the workflow without access to the entrant's personal recordings.
Three surfaces, one engine
Bee's own integration story is three doors, and the same question is worth asking through all three because the audiences arrive at different moments.
| surface | for | when |
|---|---|---|
| the dashboard | the person | after the fact, with the timeline and the evidence |
| an MCP server | an agent that speaks MCP | mid-task, before it writes the patch |
mmd + an Agent Skill |
an agent with a shell | mid-task, no transport, no session |
The second and third are the Assumption Firewall, and they are the part I would build next even if the hackathon ended tomorrow. A coding agent is handed human context constantly and has no way to tell a fact from a memory. Told "the worker retries three times, so a slow consumer isn't the problem", it writes a confident patch on a premise that stopped holding three weeks ago, and then defends it, because it reasoned correctly from what it was given.
check_assumption("The checkout worker retries three times, so a slow consumer is not the problem.")
DRIFTED -- Checkout retry attempts
stated 3, actually 1
changed 2026-08-23 -- Reduce checkout retries to 1 after duplicate-charge incident
severity HIGH
restated in 6 earlier conversation(s), 2 of them after the change
Act on the actual value, and tell the human what changed and when rather than
silently correcting them.
evidence: AWS_APPCONFIG OK appconfig://ecommerce/production/checkout-worker$.retry.max_attempts
The last line is the product. Knowing the value is not enough: an agent that silently substitutes it leaves the person still believing the old one, ready to repeat it in an hour, next to somebody else, where nothing is checking. Most of the Agent Skill is about how to say it.
The firewall lives in the engine rather than in any one surface, so a sentence gets the same verdict
from the same registry through the same gate whichever door it arrives at. Exit codes carry it:
0 supported, 1 drifted, 2 inconclusive. Three, not two, because a connector that could not be
read must never be indistinguishable from a person being wrong.
Why this is not an LLM wrapper
A model is never asked whether a statement is true. It is asked one question — which registry
property is this sentence about? — and the answer passes through a deterministic gate before any
source is read. Truth comes from AppConfig, Sentry and the repository. packages/drift-spec
contains no I/O and no model at all.
Propose, then gate. Two independent proposers feed one grounding gate that requires the subject alias, a property lexeme and the value literal to actually appear in the spoken words, in a clause that asserts rather than asks. A proposer that invents a number cannot get past it:
"The checkout worker retries a bunch of times." proposed max_attempts = 3
rejected: the asserted value does not appear literally in the utterance
Four verdicts, never two. SUPPORTED / DRIFTED / INCONCLUSIVE / UNSUPPORTED_TYPE. There
are 34 adapter tests, and every failure mode in them — timeout, 403, malformed document, absent
property, sources that disagree — produces INCONCLUSIVE.
No competency score. The system never aggregates how often a person was wrong, and there is no per-person metric anywhere in the store. A tool that could produce one would be used to produce one.
Measured, not asserted
pnpm eval --verdicts runs 204 labelled utterances: 51 supported claims, 51 drifted claims, 51
technical-but-unverifiable statements, 51 non-claims.
| grammar proposer, no model | grammar + Bedrock | target | |
|---|---|---|---|
| candidate precision | 100.0% | 100.0% | > 95% |
| recall | 90.2% | 93.1% | |
| false-positive rate | 0.0% | 0.0% | |
| subject / property mapping | 100.0% | 100.0% | > 95% |
| value extraction | 100.0% | 100.0% | > 98% |
| verdict accuracy | 100.0% | 100.0% | |
| latency | 2.9 ms / utterance | 2.2 s / utterance |
The second column is a real model on Bedrock, run over the whole corpus on 18 September. It raises
recall and does not touch precision, because both proposers feed the same deterministic grounding
gate and the gate is what holds precision. Recall is deliberately imperfect: the seven remaining
misses are the gate declining on purpose, a negated clause or a sentence with no lexeme for the
property, and each false positive avoided is a person not being told they are wrong about
something they never said. pnpm eval --errors prints them.
Building that corpus found six real extraction defects in an afternoon, including one that
inverted a claim: "the DLQ on the checkout worker is inactive" read on as a polarity word
inside a prepositional phrase. None would have been found by more unit tests written by the person
who wrote the code.
The part I am most glad I did
Integration tests run against tools/bee-sim, a test proxy
implementing the documented /v1 wire protocol. When building client tooling, the danger is having
a test harness share a misconception with the client: a test staying green while both sides share the
same misconception.
That happened, and the test proxy and the client were both written from the documentation's
examples, so they agreed on everything the examples get wrong. The examples show a stream frame with
a speaker and no type, no utterance id and no time, and conversations with their words in a flat
list. I concluded bee proxy was a subset of the CLI and built a hybrid client that shelled out for
reads the proxy "did not have". All of it was wrong, and all of it had passing tests.
What Bee actually does came from @beeai/cli's source, which is MIT and one gh repo clone away,
and from published live frames. A frame names its type on the SSE event: line and in its own type
field, and carries the utterance's id and spoken_at, but no speaker. A conversation record has a
numeric id and no uuid, keeps its words under transcriptions[] (a realtime pass, then a final one
with new ids), gives every time in epoch milliseconds and every speaker as "Unknown".
GET /v1/changes returns bare ids with its cursor at the top level; it is bee changed --json that
moves it under meta. The proxy forwards every /v1 path upstream and has no route list at all.
Against an account, the example-shaped client would have read every transcript as empty, never
advanced its cursor, and never matched a live sentence to its conversation.
So the test proxy now answers in the API's shapes, and tests/conformance/bee-wire.test.ts holds it
there: it copies parseSSEBuffer verbatim out of @beeai/cli, runs the literal wire bytes
through Bee's own parser, and pins each field of a frame, a conversation read and the changefeed.
A live sentence is joined to its conversation through the utterance id, the one thing the frame and
the record share.
pnpm doctor checks the configured connection: it runs the application client and prints a row per
capability, including whether the last realtime frame arrived named and carrying its utterance id,
and whether a transcript read returns any utterances.
How it is built
TypeScript throughout, one pnpm workspace, 244 tests, tsc --noEmit clean.
packages/drift-spec/ types · adjudication · severity · validation [no I/O, no model, no Bee]
packages/bee/ the only file that talks to Bee · event identity · fingerprints
packages/engine/ registry · speech acts · grounding · polarity · extraction · adapters
store · recurrence · pipeline · reconcile · capture · firewall
apps/server/ HTTP API, SSE, dashboard host
apps/dashboard/ drift cards, evidence, mental-model timeline, conversation review, agent checks
apps/mcp/ the Assumption Firewall over MCP
tools/cli/ `mmd`, the command the Agent Skill drives
tools/doctor/ the Bee preflight
tools/bee-sim/ wire-conformant test proxy for protocol conformance and failure injection
tools/eval/ the golden corpus and the metrics harness
skills/ the Agent Skill
infrastructure/ CDK: 35 resources
tests/conformance/ protocol wire bytes, through Bee's own SSE parser
AWS
| service | used for | where |
|---|---|---|
| AppConfig + AppConfigData | authoritative deployed configuration and feature state; hosted version history reconstructs when a value changed | packages/engine/src/adapters/appconfig.ts |
Bedrock (Claude via @anthropic-ai/bedrock-sdk, any other family via the API endpoint's chat completions) |
the second extraction proposer, asked only which registry property a sentence is about; over the corpus, recall 90.2% → 93.1% at 100% precision | packages/engine/src/extract/bedrock.ts |
| DynamoDB | single-table store: claims, evidence, drifts, cursor, dedupe markers with TTL | packages/engine/src/store/dynamo-store.ts |
| CloudWatch | the metrics that matter: how much was heard, how little was acted on, how often a card was dismissed | infrastructure/lambda/index.ts |
| SQS, Lambda, API Gateway, Secrets Manager | the deployed topology | infrastructure/cdk/ |
cd infrastructure/cdk && npx cdk synth synthesizes 35 resources and bundles the handlers with
esbuild from the same packages/ source the tests run against.
What I would build next
Team-scoped drift. Right now a belief is one person's. The interesting version is the moment two people on the same team hold different values for the same property, which Bee can see and nothing else can. That is the feature I would build first with more time.
Built With
- amazon-web-services
- appconfig
- bedrock
- bee
- bee-cli
- cdk
- changed
- dynamodb
- mcp
- node.js
- pnpm
- react
- realtime-stream
- search
- sentry
- typescript
- vite
- vitest
