Mental Model Drift

Detects when what an engineer believes about a system no longer matches what the system is, using Bee on a wearable or Apple Watch to hear the claim and the deployed configuration to check it.

We already monitor configuration drift, infrastructure drift and schema drift. This monitors the one system nobody instruments: the engineer's understanding.

Amazon Developer Hackathon — Bee track (developer experience) · AWS Builder · Open Source

Mental Model Drift has used data recorded and processed through an Apple Watch running Bee software, accessed through an authenticated bee proxy.

No private Bee conversation data is included in this repository. The checked-in tools/bee-sim provides deterministic, non-private fixtures so judges and developers can reproduce the workflow without access to the entrant's personal Bee data. Its search and Facts store are test doubles; Bee's own neural search and persistent memory are used when connected to the authenticated proxy.

Explore the dashboard: https://mental-model-drift.onrender.com — a public instance using those non-private fixtures. Open a card's View evidence for the source, the locator and the timestamp; the Conversation review tab for everything the product decided not to speak about; the Agent check tab to hand it a sentence of your own and get the verdict an agent would get. It sleeps when idle, so the first load takes a few seconds to wake.

The problem, in one sentence

"It's probably fine. Checkout retries failed jobs three times anyway."

In the reproducible checkout scenario, that was true in July. On 23 August the retry count was cut from 3 to 1 after a duplicate-charge incident. Nobody told the person wearing the Bee, because nobody tells anybody: the commit was reviewed, the config was deployed, and the change never reached the mental model of the engineer who is, right now, deciding not to investigate an alert.

Sixty seconds later the dashboard says what changed, when, which commit did it, and that the same belief has been stated in five conversations since 14 July — four of them while three was still the right answer, and this one, which is not. Two hours after that the same person repeats it to another team in a corridor, while the stream happens to be down, and the product catches that too.

Related MCP server: agentseed

What it does

1. Hears the claim through Bee's realtime new-utterance stream.

2. Decides it is a claim at all — deterministically. Questions, opinions, hypotheses, plans and beliefs the speaker has already marked as past are dropped before anything is checked.

3. Verifies it against the source the registry names as authoritative: AWS AppConfig for deployed configuration and feature state, Sentry for the running release, the checked-in schema for structural facts.

4. Explains it: reconstructs the commit that moved the value, and searches the wearer's own Bee history for every earlier time they said the same thing.

5. Corrects it: writes the verified value back into Bee's memory as a confirmed fact, and prepares a documentation pull request against the file that still says the old number.

And, most of the time, does nothing at all. Across 115 utterances in the checked-in fixture corpus, 17 contain a checkable claim; they are about five registered properties, and three of those no longer match the reference configuration. The other 98 utterances produce nothing. Silence is the feature, and the dashboard's Conversation review tab shows the survey so it is measured rather than claimed (pnpm corpus).

Four ways Bee is used, not one

packages/bee/src/client.ts is the only file that talks to Bee. It uses the documented surface and nothing else: bee proxy's /v1/* endpoints and its SSE stream, with the bee CLI as the fallback for machines with no proxy running.

Two details in there are worth a reviewer's thirty seconds, because both were bugs first:

• A realtime frame's type is its SSE event: name. Bee's own client discards any frame that arrives without one, and three real event types have payloads that a shape-guesser reads as each other. The name is now read as authoritative, and anything inferred is marked as inferred.

• bee proxy is a transparent pass-through, not a subset of the CLI. Every /v1 path is forwarded upstream, so GET /v1/conversations/:id/related works over the proxy and no capability quietly disappears on a machine without the CLI installed.

Neither is stated in Bee's published documentation; both came out of reading @beeai/cli's source, and both are written up in docs/friction-log.md.

Quick start

Requires Node.js 22+ and pnpm 9.15.9. Install and build from the repository root:

pnpm install
pnpm dashboard:build

Connect your Bee account

Record a conversation through your Bee device or Apple Watch running Bee software, then start the authenticated proxy in one terminal:

npm i -g @beeai/cli
bee login
bee proxy

In a second terminal, run the server against that proxy. Keep personal derived state separate from fixture state:

BEE_PROXY_URL=http://127.0.0.1:8787 BEE_ALLOW_CLI=0 MMD_STATE=.bee-account/store.json MMD_STREAM=1 pnpm server

For PowerShell:

$env:BEE_PROXY_URL = 'http://127.0.0.1:8787'
$env:BEE_ALLOW_CLI = '0'
$env:MMD_STATE = '.bee-account/store.json'
$env:MMD_STREAM = '1'
pnpm server

Open http://127.0.0.1:4310. pnpm doctor checks the configured Bee connection; pnpm corpus --bee reads recorded conversations directly through it. Facts writes happen only after the user confirms ownership and chooses Update my understanding. pnpm doctor --write separately exercises a diagnostic Facts write.

Point MMD_REGISTRY at the properties you want checked and configure the corresponding source adapters; see source-registry.md. Bee transport and source configuration are independent: MMD_MODE=live selects live AppConfig, GitHub and Sentry adapters, and each can be selected separately as described below. The defaults use the checked-in source fixtures; run pnpm demo:seed once to create the checkout reference repository if you use that example registry.

Unset BEE_PROXY_URL and set BEE_ALLOW_CLI=1 to use the authenticated bee CLI. The dashboard reports the configured transport.

Reproduce without personal data

pnpm demo runs the engine and dashboard with the deterministic fixture proxy on http://127.0.0.1:4310. It exercises extraction, verification, recurrence, Facts interactions and disconnect recovery without a Bee account. It does not capture wearable audio or modify a Bee account. Use a separate terminal so personal environment settings are not inherited by the fixture runner.

Why this is not an LLM wrapper

A model is never asked whether a statement is true. It is asked one question only — which registry property is this sentence about? — and its answer is then passed through a deterministic gate before any source is read. Truth comes from AppConfig, Sentry and the repository.

Two independent proposers. GrammarProposer is registry-driven with no model at all; BedrockProposer is Claude on Bedrock. Agreement between them is recorded as corroboration and worth +0.07 confidence. The grammar proposer is never dropped when Bedrock is available: it is the corroborating second opinion, the offline path, and the measurement baseline.

Every candidate must be grounded in the words that were spoken. The subject alias, a lexeme for the property and a literal for the value all have to appear in the utterance, in a clause that asserts rather than asks. A proposer that invents a number cannot get past it:

"The checkout worker retries a bunch of times."   proposed max_attempts = 3
  rejected: the asserted value does not appear literally in the utterance

Four verdicts, never two. SUPPORTED / DRIFTED / INCONCLUSIVE / UNSUPPORTED_TYPE. A binary true/false forces a connector failure to masquerade as drift, which is the single worst thing a product that tells people they are wrong can do. There are 34 adapter tests, and every failure mode in them — timeout, 403, malformed document, absent property, sources that disagree — produces INCONCLUSIVE.

Measured, not asserted

pnpm eval --verdicts

204 labelled utterances: 51 supported claims, 51 drifted claims, 51 technical-but-unverifiable statements, 51 non-claims (questions, opinions, hypotheses, directives, past beliefs, reported speech, small talk).

Recall is the number deliberately left imperfect. The ten misses are phrasings outside the registry's declared vocabulary and values too far from their property lexeme to be trusted; with a model beside the grammar, seven remain, and every one of those is the grounding gate declining on purpose. Each is a missed opportunity, and each false positive avoided is a person not being told they are wrong about something they never said. pnpm eval --errors prints them.

Building that corpus found six real defects, all now fixed and pinned by tests: on/off read as polarity words inside prepositional phrases (which inverted a claim), a number regex that lost every value spoken at the end of a sentence, negation that only looked backwards so "has no user agent column" read as the opposite, a unit-blind reader that turned "backs off five seconds" into a retry count of five, deontic "should" scored as an assertion, and a legitimate negated assertion being dropped. See docs/friction-log.md.

Run everything

pnpm verify        # typecheck, 226 tests, evaluation harness
pnpm test          # 226 tests: unit, adapter failure matrix, 12 golden scenarios, MCP,
                   #            wire conformance, the corpus gate, the server over real HTTP
pnpm doctor        # Bee preflight; add --write to exercise a diagnostic Facts write
pnpm eval          # extraction metrics against the golden corpus
pnpm corpus        # dry-run the registry over the demo conversations: what would this speak about?
pnpm corpus --bee  # ...the same, read straight from `bee proxy`
pnpm server        # the pipeline and dashboard, using your configured Bee account
pnpm demo          # deterministic reproduction with non-private fixtures
pnpm mcp           # the Assumption Firewall over MCP

pnpm doctor is the preflight. It runs the client over the configured transport and prints one row per capability — including whether the last realtime frame arrived carrying its SSE event name, which is the difference between reading Bee's stream and guessing at it.

The Assumption Firewall (MCP)

The dashboard is for the person. The MCP server is for the agent sitting next to them.

A coding agent is handed human context constantly and has no way to tell a fact from a memory. Told "the worker retries three times, so a slow consumer isn't the problem", it will write a confident patch on a premise that stopped holding three weeks ago, and defend it, because it reasoned correctly from what it was given.

claude mcp add mental-model-drift -- npx tsx apps/mcp/src/main.ts

check_assumption("The checkout worker retries three times, so a slow consumer is not the problem.")

  DRIFTED -- Checkout retry attempts
    stated 3, actually 1
    changed 2026-08-23 -- Reduce checkout retries to 1 after duplicate-charge incident
    severity HIGH
    restated in 6 earlier conversation(s), 2 of them after the change
    Act on the actual value, and tell the human what changed and when rather than
    silently correcting them.
    evidence: AWS_APPCONFIG OK appconfig://ecommerce/production/checkout-worker$.retry.max_attempts

Five tools: check_assumption, belief_history, list_verifiable_properties, open_drifts, record_understanding. The last one writes to Bee memory and is off unless MMD_MCP_ALLOW_WRITES=1; it also refuses any belief that could not be attributed to the wearer. 14 tests drive it through a real MCP client.

Three surfaces, one engine

Bee's own integration story is three doors — the CLI, MCP, and Agent Skills — and the same question is worth asking through all three, because the audiences are different and they arrive at different moments.

mmd check "the checkout worker retries three times, so a slow consumer isn't the problem"
# DRIFTED -- Checkout retry attempts ...
echo $?   # 1

The exit code is the contract: 0 supported or nothing checkable, 1 drifted, 2 inconclusive. Those are three codes rather than two on purpose — a connector that could not be read must never look like a person being wrong, and an agent trusting != 0 would conflate them. tests/e2e/cli.test.ts asserts each from outside the process, where an agent would see it.

skills/mental-model-drift/SKILL.md is the Agent Skill: it composes with bee-computer/bee-skill over the same bee login session, and most of it is about how to say it. Silently substituting the right number is the worst outcome — the person keeps the old one and repeats it in an hour, next to someone else, where nothing is checking. And they were usually not wrong: four of the five earlier times they said "three retries", three was the correct answer. The software moved. Nobody told them. The skill's job is to get an agent to say that, in one sentence, and then carry on with the real value.

AWS

cd infrastructure/cdk && npx cdk synth synthesizes 35 resources and bundles the handlers with esbuild from the same packages/ source the tests run against.

Each source is switched on its own. MMD_MODE=live turns all three live; MMD_APPCONFIG, MMD_GITHUB and MMD_SENTRY take live or local and override one, so the deployed configuration can be read from AWS while the repository is the local clone:

AWS_REGION=us-east-1 MMD_APPCONFIG=live pnpm mmd check "the checkout worker retries three times"
MMD_DYNAMO_TABLE=mmd-dev pnpm demo                        # the store is the table
MMD_BEDROCK_MODEL_ID=openai.gpt-oss-120b pnpm eval --proposers grammar,bedrock

The first two lines and the third were run against the real services on 18 September; docs/product-feedback.md quotes what came back, and friction-log entries A4 to A6 are the four defects that running them found.

Layout

packages/drift-spec/      types · adjudication · severity · registry validation   [the OSS artifact]
packages/bee/             Bee client · event classification · fingerprints        [the track hook]
packages/engine/          registry · speech acts · grounding · polarity · extraction
                          adapters (appconfig, github, sentry) · store · recurrence
                          pipeline · reconcile · capture · docs-pr · config
apps/server/              HTTP API, SSE, dashboard host
apps/dashboard/           drift cards, evidence panel, mental-model timeline, conversation review
apps/relay/               the local process that sits next to `bee proxy`
apps/mcp/                 the Assumption Firewall
tools/bee-sim/            wire-conformant test proxy for protocol conformance and failure injection
tools/doctor/             the Bee preflight: every capability, over the live transport
tools/cli/                `mmd` -- the command the Agent Skill drives; verdict in the exit code
skills/mental-model-drift/    the Agent Skill, composing with bee-computer/bee-skill
tools/eval/               the golden corpus and the metrics harness
tools/demo/               fixture runner, corpus audit, reference repository seeding
infrastructure/           CDK stack and Lambda handlers
tests/conformance/        wire format verification through Bee's own SSE parser
tests/                    226 tests

packages/drift-spec has no I/O, no model and no Bee: it is the portable half, and it is the piece intended to be useful to anyone building this kind of verification for a different source.

Documentation

• docs/architecture.md — the pipeline, and why each boundary is where it is

• docs/privacy.md — what is stored, what is never stored, and what leaves the machine

• docs/threat-model.md — including the ways this product could hurt someone

• docs/source-registry.md — how to point it at your own systems

• docs/friction-log.md — building against Bee, AWS and MCP: what worked, what did not

• docs/conformance.md — wire format verification against Bee's protocol

• docs/product-feedback.md — the submission's feedback answers

Licence

Apache-2.0. See LICENSE.