ENGRAM

Inspiration

On-call engineers get paged for the same missing-index regression over and over, and most "AI ops" tools either hallucinate a fix or apply one with no memory of whether it worked last time. We wanted an agent that behaves like a reliability engineer with a notebook: reason about a regression using the database's own query plan, ask a human before touching anything, then remember the fix well enough to recognize the same problem faster the second time. CockroachDB's VECTOR type and distributed C-SPANN index made "it remembers" a literal, measurable claim a similarity score on screen instead of a marketing word.

What it does

Engram watches a CockroachDB cluster and, when a query regresses, runs a five-node loop: Observe → Recall → Reason → Gate → Act & Measure. It recalls similar past incidents from its own dedicated memory cluster, proposes a fix using the target cluster's real EXPLAIN plan, stops for one human approval, checks CockroachDB Cloud's own backup API before touching anything, applies the fix, and measures the real before/after latency. Two things it proves on screen: (1) a second incident against a familiar query resolves with a visible similarity/confidence score, not narrated; (2) killing the agent's AWS ECS Fargate task mid-fix, with no warning, produces a clean resume from a real LangGraph checkpoint and exactly one remediation row recorded never two.

How we built it

Reasoning: Ollama Cloud (minimax-m3:cloud). Embeddings: Cohere (embed-english-v3.0, natively 1024-dim). Amazon Bedrock's InvokeModel is blocked account-wide on this AWS account confirmed empirically, not assumed so neither the reasoning nor the embeddings run on Bedrock.
Memory: a second, dedicated CockroachDB Cloud cluster (structurally separate from the one Engram operates on), using VECTOR(1024) + a C-SPANN vector index for real semantic recall, hybrid-scored (similarity · confidence · recency · entity affinity), Row-Level TTL for forgetting, and a Wilson-lower-bound confidence score so a lucky 1/1 fix never outranks a proven 47/50 one.
Agent: a 5-node LangGraph loop running on AWS ECS Fargate aws ecs stop-task is the resilience demo, not a simulation.
AWS glue: SQS (FIFO, deduplicated by incident fingerprint) + EventBridge trigger the agent; Lambda runs the memory-lifecycle workers (consolidation, decay, embedding backfill) and the approvals/metrics/webhooks API separately from the agent, so memory upkeep survives even if the agent doesn't; S3 holds task artifacts; Secrets Manager plus four least-privilege IAM identities keep the blast radius small.
Dashboard: a read-only Next.js app on Vercel, server-sent events over a dedicated read-only SQL role no DB credential or API key ever reaches the browser.
We used CockroachDB's Managed MCP Server during development to empirically measure its real limits (response-size ceiling, query timeout, the implicit LIMIT 25, deny-listed system schemas) those measurements are why the production agent's hot-path reads go through psycopg3 instead, leaving MCP as the control-plane, capability-discovery tool it's actually designed for.

Challenges we ran into

Amazon Bedrock's InvokeModel came back blocked account-wide partway through the build a same-day pivot to Ollama Cloud + Cohere behind the same provider abstraction, so the swap was a config change, not a rewrite.
A transparent network proxy blocked outbound :26257 locally for most of the build worked around with GitHub-hosted CI runners for migrations and a client VPN for interactive sessions, never assumed to still be open the next day.
A real, previously-latent bug only surfaced once we actually ran the "it remembers" demo for real: recalling a memory item with a not-yet-backfilled NULL embedding crashed the hybrid scorer (0.45 * None) instead of being excluded from ranking found live, fixed, redeployed.
CockroachDB checks SELECT privilege on a nullable foreign key's referenced table even when that column is omitted from an INSERT a real, previously-unknown-to-us privilege-check behavior that broke a least-privilege Lambda role before we ever ran it for real.
A hard RU budget: our own live smoke-testing burned roughly half of the CockroachDB Cloud free tier's request-unit pool across the build, forcing deliberately smaller (thousand-row, not million-row) scratch tables in the final days before submission.

What we learned

That "the agent remembers" is either a real, falsifiable claim you can point a skeptic at a similarity score on screen, a row count that stays at exactly one after a kill or it's marketing. Building toward "how would a skeptic disprove this" produced a more honest project than building toward "does the demo look good."

What's next

Live traffic discovery via the MCP server's query-introspection tools instead of an operator-maintained watch-list; enabling the already-built (but currently paused, cost-conscious) consolidation/decay lifecycle workers; and a true LangGraph checkpoint-skip on resume, since today's kill-and-resume proof relies on database-level idempotency rather than skipping already-completed graph nodes.