Apprentice — memory that helps, then hurts

• 

  Oracle rises to 88%. Both Bedrock agents peak before full memory and decline. Retrieval at that last epoch is 44/44.

• 

  Same exam, earlier brain. AS OF SYSTEM TIME — not a backup restore.

• 

  Four doors into one memory (CLI, exam, MCP, Lambda). CockroachDB is the brain. Bedrock generates; Lambda runs the turn. Warehouse is a prop.

• 

  Teach → distill → store → rewind → answer → grade. The full agent loop.

• 

  The memory plane, live: CockroachDB Cloud on AWS, Frankfurt (eu-central-1)

• 

  Real Bedrock calls: Titan + Nova, us-east-1 (what it calls)

Inspiration

I ask an analyst agent for regional revenue. It writes competent SQL — and gets my company's rules wrong:

• uses the calendar year when I said fiscal
• counts a soft-deleted order
• attributes region through the product instead of the customer

Tomorrow someone corrects it. The week after, the same mistake comes back — because nothing in the stack can show what the agent knew when.

Most "agent memory" demos are a chat that got longer and a curve that only goes up. I wanted three things instead: a frozen exam, a brain I can rewind, and permission for the curve to fall.

If memory cannot be replayed, it is not memory. It is a log. If it cannot get worse, you are not measuring it.

CockroachDB's AS OF SYSTEM TIME is that rewind. Not a backup restore. A timestamp.

For judges — fastest paths in

Path	Link	Time
Ask the agent yourself — live	prasadt1.github.io/apprentice-crdb ↗	60 s
Watch the demo	youtu.be/xmxEFkEiJLY ↗	2:59
Source	github.com/prasadt1/apprentice-crdb ↗	30 s
Published curve, with every limitation	eval/RESULTS.md ↗	60 s
Proof package — every claim → its receipt	docs/proof/PROOF.md ↗	90 s
Verify the freeze yourself	pytest -q then python eval/run_exam.py verify	~1 min

TL;DR: Freeze a 50-question exam → teach into CockroachDB → score each AOST epoch. Oracle rises to 88%. Two Bedrock agents peak on partial memory and decline. At full memory, retrieval is 44/44 and accuracy is falling. Storage and recall look perfect. Utilization does not.

You can run this yourself — it is not a slide deck. On the hosted page, pick any of the fifty frozen questions and ask it twice: once with an empty memory, once after recalling the house rules from the live CockroachDB cluster. Real recall, real Amazon Bedrock generation, real execution against the warehouse, graded against labels frozen before any of it existed. About five seconds per call. Start with q02 — cold it assumes the calendar quarter and answers 20,000; with memory it shifts to May–July, nets refunds, and answers 95,000.

I am not shipping a warehouse copilot. CockroachDB is not required to generate SQL. It is required to store and replay the education — the receipt I cannot edit, including the epoch where more memory made the agent worse.

Primary user: the analytics engineer who owns the house definitions. Secondary: the reviewer who needs to see what the agent knew last Tuesday. Integration model: CLI + CockroachDB Cloud + Bedrock + Lambda (hosted demo) — not a BI fork.

Who this costs, and what it costs them

Agent memory is measured on two legs — stored, recalled. Both were perfect at the exact epoch my agent got worse. If you run Mem0, Zep, LangChain memory or something homegrown, your dashboard stays green through that. Silent regressions ship.

Who pays, concretely:

• The analytics engineer — answers "revenue excludes refunds" for every new hire, and now for every AI tool on the team. When the agent gets it wrong the number lands in a board deck, not an error log.
• The reviewer signing off — cannot see what the agent knew when it answered. There is no git blame for a memory.
• Everyone already paying for this — dbt's semantic layer and Looker's modelling layer are industries built because house rules live in people's heads. An agent is one more consumer, except it cannot be onboarded and never says it is unsure.

The shape of the fix is a release gate for agent memory: freeze an exam of your own house rules, replay it at every memory state, refuse the change if utilization dropped. Same reflex as refusing a deploy on a failing test.

None of this is SQL-specific — any agent accumulating instructions, preferences or policy hits the same question, and today almost nobody can answer it. The warehouse here is a prop. The instrument is not.

What it does

Apprentice teaches five house rules into CockroachDB — soft-delete, cancelled orders, net revenue, fiscal calendar, region via the customer — then replays a frozen 50-question execution-graded exam at each stage of that education.

Four compressed epochs (live GC TTL on Basic is 75 minutes, so the clock is minutes, not months):

1. memory_zero — empty brain.
2. filters — deleted_at IS NULL, status <> cancelled.
3. revenue_and_fiscal — net of refunds, FY starts 1 February.
4. join_path — region through customers, not product_id.

Six of the fifty items are published regression baits — places where a plausibly over-generalized rule should make the agent worse. What actually happened is in The numbers below.

Oracle 6-7-31-44. Nova Micro 8-21-18-13. Nova Lite 14-19-19-16. Bold peaks are not at full memory. Retrieval at join_path is 44/44.

One correction, on camera

The demo runs this live against the cluster, in six titled sections. Fiscal Q2 net revenue, from the frozen exam:

• Cold, zero rules retrieved: Nova Lite assumes the calendar quarter and answers 20,000. The exam grades it wrong.
• One correction. sqlglot diffs the fix against the attempt as syntax trees — no model on that path — and distils four rules: the fiscal calendar, refund netting, and two order-hygiene filters. They land in CockroachDB under a new timestamp.
• Same question, same model. Recall returns those rules, the window shifts to May–July, refunds come off, and the answer is 95,000. Correct.
• Rewind. Same cluster, two timestamps: four rules now, zero at t0 — because none existed then. A clause, not a restore.

I picked this question because its cold failure is structural: hitting 95,000 requires knowing the fiscal year starts 1 February and netting refunds — a model cannot luck into it. (Why that matters is in Challenges below.)

Replay is a timestamp clause. I can show the moment before the bait fired.

The numbers

I froze the exam (b043aea) before any agent existed. Labels were not edited. Everything below is execution-graded. Full write-up: eval/RESULTS.md.

Epoch	Live rules	Oracle (ceiling)	Nova Micro	Nova Lite
memory_zero	0	6/50 · 12%	8/50 · 16%	14/50 · 28%
filters	2	7/50 · 14%	21/50 · 42%	19/50 · 38%
revenue_and_fiscal	4	31/50 · 62%	18/50 · 36%	19/50 · 38%
join_path	5	44/50 · 88%	13/50 · 26%	16/50 · 32%

Bold = that column's peak. Both agents peak before full memory. Lite is better cold and still ends below its own peak — so this is not Nova Micro being weak.

The oracle is not an agent. It is a selector over frozen SQL bodies, gated on exam metadata. Its 44/50 cannot fail and never wrote a line of SQL. I keep it only as the ceiling: what a perfect consumer of the retrieved rules would score. I briefly published that 44/50 as the learning curve. That was wrong. The correction is why the agent columns exist.

Retrieval was not the bottleneck. At join_path, every rule an item needed was in the model's context on 44/44 rule-bearing items. k=5 was fixed before the run and never adjusted after a score. With five live rules that means retrieval returns the whole corpus and ranks nothing — every prompt carries all five rules, including on unaffected items that need none.

Over-application is real, and not monotonic with rule count. Both models end below their own unaffected peak — Micro 5/6 → 3/6, Lite 6/6 → 4/6 — but Lite's worst cell is at two rules, not five. Per-stratum tables in eval/RESULTS.md.

My own prediction failed, and I published it. I froze six regression baits expecting all six to fire; Lite resisted four at full memory. One bait turned out to measure a model prior rather than a memory regression, and another flipped entirely — memory corrected a bias I had predicted it would cause.

The 56-point gap at full memory is entirely could the consumer use what it retrieved.

Reproduce it

Judges do not need my cluster or an AWS account to check the freeze:

git clone https://github.com/prasadt1/apprentice-crdb.git && cd apprentice-crdb
pip install -e ".[dev]" && pytest -q && python eval/run_exam.py verify

Expect VERIFY OK with sha256s matching eval/FREEZE.md. Live-memory and Bedrock instructions are in the README and the testing instructions; every generated SQL string from both arms is already committed under eval/runs-*/.

How I built it

CockroachDB is the memory layer — rules, episodes, vectors, AOST. Bedrock generates; AWS Lambda executes the hosted turn. The warehouse is a prop. One binder for teaching; one answering path that cannot see the labels.

Beyond what the diagram shows:

• Freeze first — eval/questions.json and eval/labels.json are immutable after FREEZE.md. Wrong label → erratum in RESULTS, never an edit.
• Two curves on purpose — oracle is the ceiling (did memory hold the rule). Agent is the measurement (could the model use it). The A−B gap is the finding.
• Hard leakage rule — the generator consumes {id, question} plus warehouse DDL and retrieved rule statements. Never house_rules.py, never seed.sql, never gold/naive/bait.
• Vector index is real; the planner does not use it here — CREATE VECTOR INDEX semantic_embedding_idx succeeded (vector_l2_ops). Live EXPLAIN of the answering-path recall still reads semantic_rules@semantic_rules_pkey / spans: FULL SCAN (15 estimated rows, 100% of the table, 5 live rules). The index exists; the corpus is too small for the planner to pick it. I say so rather than claim scale I do not have. Titan v2 is 1024-d (not 384); the column was widened to match.
• Compressed clock — Basic will not raise gc.ttlseconds above 4500. Epochs are minutes apart. I disclose that instead of pretending I have months of history.

What happens when things go wrong

• Concurrent teachers — rule upserts are SERIALIZABLE; one live row per key; the old row is superseded, not deleted.
• Time-travel races — I wait for the closed timestamp, then BEGIN AS OF SYSTEM TIME <hlc>. I do not BEGIN at now and then AS OF.
• Bad SQL — a reject grades wrong, never skipped.
• Untrustworthy self-report — at memory_zero, with zero rules retrieved, Nova Micro still emitted -- used_rules: on 34/50 items. Only the retrieval log and the graded result count.
• Blast radius — MCP is read-only and never called at answer time; the hosted demo's Lambda uses a read-only SQL user, a fixed question set, and a daily cap.

CockroachDB tools used

Tool	How, not just that it is configured
Distributed vector indexing	CREATE VECTOR INDEX succeeded; answering path orders by <->. EXPLAIN is a full primary scan — five rows, planner will not use the index. Receipt in eval/RESULTS.md
Cloud Managed MCP Server	Cursor, read-only, to inspect the live schema and row counts while I built. The agent does not call MCP at answer time.
ccloud CLI	Official binary. Raw ccloud cluster list -o json output committed at docs/proof/ccloud-cli.json — the live Basic cluster on AWS eu-central-1, v26.2.5. A Cloud API listing of the same control plane is kept beside it as a secondary receipt.

AWS services used

The cluster is CockroachDB Cloud on AWS (eu-central-1). That satisfies “deployed on AWS,” and I do not count it as an AWS service I built with.

Service	How
Amazon Bedrock	Titan Text Embeddings V2 on write and query; Converse API for generation. Pre-registered model: amazon.nova-micro-v1:0. Replication arm (added after seeing the Micro drop, disclosed): amazon.nova-lite-v1:0. Calls run in us-east-1 (on-demand).
AWS Lambda	Serverless agent execution for the hosted demo. A Function URL runs the full turn — Titan embed → embedding <-> recall against the live cluster → Converse → execute → grade — so judges operate the real agent, not a replay. Read-only SQL user, fixed question set, per-IP rate limit, daily cap, CORS locked to the Pages origin; degrades to the recorded run if unavailable.

Challenges I ran into

• AS OF on the SELECT raced psycopg’s implicit BEGIN (“inconsistent AS OF SYSTEM TIME timestamp”). The fix is BEGIN AS OF SYSTEM TIME <hlc> on an autocommit connection, after waiting for the closed timestamp.
• Titan v2 does not do 384 dimensions, and CREATE TABLE IF NOT EXISTS will not widen an existing column — the ALTER then fails its backfill against rows of the old width, surfacing only in the post-commit phase. Clearing the stale vectors first fixed it.
• The first educate run plateaued at 15/50 because the distiller named joins join:customers. I did not edit labels. I fixed the keys and re-ran.
• I published the oracle 44/50 as the learning curve, then had to take it back. The agent curves exist because of that mistake.
• The first demo question failed cold only by luck — a later run answered it correctly cold. I swapped it for one whose failure is structural, because a demo that depends on a model failing on cue is not a demo.

Accomplishments that I'm proud of

• Exam frozen before any graded learning run — same discipline as Premortem
• A finding that replicated across two model tiers
• Publishing my own failed bait prediction instead of deleting it
• Taking back the 44/50 “learning curve” in public rather than leaving it up

What's next for Apprentice

• Relevance-gated recall (smaller k, or a filter that does not inject unused rules) — this exam is the instrument
• Let the hosted demo teach, not just ask: a judge supplies the correction and watches their own rule land in CockroachDB under a new timestamp
• A corpus large enough that the vector index actually beats a scan, so the distributed part of distributed vector indexing is doing measurable work

More evidence

One page, every claim → its receipt: docs/proof/PROOF.md — built for the "meaningfully integrated, not just initialized" bar. What it links:

• Vector index — the live EXPLAIN (real index; planner scans at 5 rows, stated not hidden)
• ccloud CLI — raw ccloud cluster list -o json from the official binary
• Managed MCP — what it inspected, and the dev-side / read-only boundary
• Amazon Bedrock — run manifests (model + region) and the CloudWatch invocation metrics
• AWS Lambda — the live hosted-turn response, with its guardrails
• The freeze — the sha256s, checkable in one command

Standalone if you want the raw files: freeze record · exam protocol · ccloud-cli.json.

Honest scope. The warehouse is six orders by design — labels are hand-checkable. Single run per model at temperature=0. Two Nova-family models only (Claude on Bedrock needed an inference profile I did not have). Nova Lite was added after seeing the Micro drop, as a disclosed extra arm, not a replacement. k=5 was fixed before the run. The unaffected stratum is n=6. Epochs are minutes apart inside a 75-minute GC window. Every generated SQL string is in eval/runs-nova-micro/ and eval/runs-nova-lite/.

Built With

• amazon-web-services
• aws-lambda
• bedrock
• cockroachdb
• mcp
• psycopg
• pydantic
• pytest
• python
• sqlglot