Memory Health Monitor

Agents don't crash when their memory breaks, they quietly answer with less. This watches agent memory on CockroachDB, names which of five failures is happening, and traces it.

Comment

Inspiration

I built an agent last week and it stopped working and nothing told me. Very frustrating as you can imagine.

There was no error, no failed query, no alert. It just started answering with less, and it sounded exactly as sure of itself as when it had been right. The health check was green the whole time, because from the database's point of view nothing was wrong. The rows were there. The queries came back. The memory just resolved to nothing useful, and the first signal I got was noticing the answers had gotten worse.

That's the gap. Memory can fail in ways the database has no opinion about.

What it does

It watches an agent's memory and tells you which of five things is going wrong.

Staleness is a row written a minute ago carrying a fact that was true a year ago. Looks fresh, isn't. Eviction pressure is memory about to expire. Retrieval degradation is when the search finds candidates but none are relevant enough to use. The memory is there, the relevance is gone. Threshold calibration is the opposite: relevant memories that the cutoff wrongly threw away. And vector integrity catches embeddings that are missing or malformed, rows that look fine but can't be searched.

Those middle two are the interesting pair. They produce the same symptom, a retrieval that comes back with nothing, but they're opposite problems. One means nothing relevant existed. The other means something relevant existed and got discarded. Fix the wrong one and you've made things worse.

It also connects memory failures to the answers they ruined, so a bad response traces back to the exact retrieval behind it.

How I built it

CockroachDB holds both the agent's memory and the record of watching it. Five tables: memory records with 1024-dimension embeddings, a retrieval log, health check results, per-agent config, and agent turns. The distributed vector index handles semantic recall, so there's no separate vector store to keep in sync. Row-level TTLs prune the observability tables on their own, which means the layer manages its own retention instead of needing a cleanup job.

On AWS, Bedrock runs Titan Text Embeddings V2 for the vectors and Claude Sonnet 4.5 as the agent's model. Lambda runs the five checks. EventBridge Scheduler fires it every five minutes. A second Lambda keeps the demo populated on a 30-minute cadence, guarded so it can only touch rows prefixed demo-.

The dashboard is FastAPI plus a single page. It's read-only, but not by convention. Every session opens with default_transaction_read_only = on, so CockroachDB refuses writes no matter what the process asks for.

Challenges I ran into

The tool found a bug in my own config. I picked a similarity cutoff of 0.35 by educated guessing. The calibration check measured every retrieval and found nine correct answers being thrown out, one of them missing by 0.0266. The obvious move was to lower the number. The better move was noticing that a hardcoded threshold quietly discarding correct answers is the failure class this whole project is about. So it became the fifth check, and I left the bad default in place so the layer demonstrates catching itself.

Building an honest signal is harder than building a green one. The natural version of this dashboard goes green when the queries succeed, which is the exact lie that started the project. So no health data means unknown, never healthy. An empty window isn't a passing grade. A failed fetch doesn't keep showing the last good render. When the frontend originally used Promise.all across seven feeds, one transient 404 threw away six successful responses and blanked the whole board, which contradicted everything else the project argues. It degrades per panel now.

CockroachDB's now() is the transaction timestamp. Every row written in one transaction gets a byte-identical value. Four agent turns written together were unorderable, and a "grab the row I just inserted" lookup couldn't identify anything. This bit me three separate times before I switched to clock_timestamp() and RETURNING id.

Telling a real valley from a wide gap. The calibration check suggests a better cutoff by finding the empty space between the discarded and returned score clusters. My first version only tested how wide the gap was, which meant a single outlier sitting far from one cluster would produce a confident recommendation for a distribution that had no second cluster at all. It now requires the gap to be wide absolutely, wide relative to the typical spacing, and to have real mass on both sides. Otherwise it draws nothing and says why.

Accomplishments that we're proud of

The two checks that read the same nineteen rows and reach opposite, both-correct diagnoses. That's the thing I'd point at first.

The honest signal. It would have been easier to ship a dashboard that shows green when the queries work. Everything in the design pushes the other way: the hero verdict refuses to go green without positive evidence, the empty states say an empty window is not a healthy one, and a partial outage marks the panel that failed rather than throwing away the six that succeeded.

The demo also breaks itself for real. Nothing is simulated. The harness expires actual rows and runs actual off-topic queries, and it's guarded so it can only touch demo agents.

What we learned

Measure the threshold, don't guess it. I would have shipped 0.35 and never known what it was costing me.

The harder lesson is that an observability tool that can't say "I don't know" isn't observability. Most of the design work here went into making the thing refuse to claim health it couldn't prove, which meant deliberately building a dashboard that shows less confidence than the naive version would.

What's next for Memory Health Monitor

Per-agent threshold recommendations that can be applied with a rollback window instead of just suggested. Alerting on the leading indicator, average similarity drifting toward the cutoff, rather than waiting for the failure itself. And packaging the five checks as a CockroachDB Agent Skill so they're usable outside this project.

Built With

Share this project:

Updates

Submission history