StdMemory

Your AI tutor's memory can't afford to forget a student mid-lesson. CockroachDB gives it memory that survives node failures—zero data loss, no stale reads, proven live.

Comment

Inspiration

Agentic systems don't behave like humans clicking and waiting — they spawn autonomously and write memory constantly, on every turn. Most databases were built for human-scale traffic, so a node failure means downtime or lost writes. On Kubernetes, a pod getting rescheduled isn't rare — it's routine. We realized the real danger isn't an agent that stops when memory breaks; it's an agent that keeps confidently acting on a version of the user who no longer exists. We wanted to prove — not just claim — that CockroachDB closes that gap, using an AI tutor as a relatable, human stage for a database story.

What it does

StdMemory is an AI tutor whose entire memory — every student interaction, every subject pivot, every embedding — lives in a self-hosted, 3-node CockroachDB cluster on Amazon EKS. A live control room lets you run scripted-but-real demo scenarios: a student pivoting subjects mid-conversation while the memory follows atomically; the same pivot happening while we kill a live database node on camera; hundreds of simulated students hammering the database while one real student still gets instant replies; and a student returning in a brand-new session while the tutor still recalls what they struggled with. Every scenario measures itself — rows lost, stale reads, replication health — live, from the database.

How we built it

An append-only memory_event log with a per-student CockroachDB vector index (C-SPANN) stores memory and its embedding in one transaction, so they can never desync. A serializable "drift transaction" handles subject pivots atomically with read-your-writes verification. A multiprocessing load harness proves survivability by hammering the cluster with synthetic students while a real interactive session runs alongside — and we validate it two ways: dynamically, by killing a real node mid-storm and confirming zero data loss; and statically, using a real CockroachDB Agent Skill that checks replication health. The whole stack — FastAPI backend, React control room, and the tutor — ships as one Docker image, deployed on self-hosted CockroachDB + EKS.

Challenges we ran into

CockroachDB's vector index needs a cluster setting enabled before schema creation, which can't run inside the same transaction as the rest of the schema — a subtle ordering bug that took real debugging to isolate. On EKS, a missing default StorageClass left our pods stuck in Pending for hours with a misleading error; once fixed, a botched pod recreation left three CockroachDB nodes briefly split across two different cluster identities, forcing a full reset. We also hit the classic new-AWS-account wall: Bedrock access gated behind account verification with no clear timeline — so we architected the whole system to run fully local-first, so AWS gaps never blocked development.

Accomplishments that we're proud of

We didn't just claim survivability — we measured it. Our load harness ran a real node kill mid-storm and reported 12,288 acknowledged writes, 12,288 found in the database, 0 rows lost, 0 stale reads. We validated that result two independent ways: dynamically (the kill test) and statically (a real CockroachDB Agent Skill confirming 3x replication on our live cluster). And we held a hard line throughout: every demo scenario scripts only the input — the student's turns — while every consequence runs through the real agent and real database, with a public/local safety boundary so the chaos-trigger scenario never risks a live deployment.

What we learned

Distributed systems failures are rarely clean — a "stuck" Kubernetes pod can hide two or three compounding root causes (missing storage class, then a split-brain cluster identity) that only surface one at a time. Also that with this persistent memory gaps can be made nearer.

What's next for StdMemory

Extend the CockroachDB Agent Skills integration with a second skill (cluster health review) for broader operational coverage. Add true multi-region deployment to demonstrate memory surviving not just a node failure but a full region outage. And bring the "no maintenance window" claim to life with a live online schema migration running under active load, as a fifth proof scenario. Also increase student parameters no just limiting to basic stuff.

Built With

Share this project:

Updates