-
-
DB
-
dashboard
Inspiration
Every team that runs a database has lived this cycle: a query gets slow, someone gets paged, they run EXPLAIN ANALYZE, spot a missing index or stale statistics, fix it, and move on — until the exact same class of problem shows up again three weeks later on a different table, and the whole diagnostic process starts from zero. We wanted to build something that actually remembers. Not a dashboard that shows you the problem, an agent that fixes it and gets faster every time it sees the same failure mode again.
What it does
AgentOps is a multi-agent system that watches a CockroachDB-backed application, detects performance incidents, diagnoses the root cause using an LLM plus vector-recalled memory of past incidents, applies a fix, verifies it actually helped, and explains what it did in plain English.
Five agents work together:
- Monitor — polls query latency and EXPLAIN ANALYZE plans, and pulls CockroachDB Cloud cluster-level context via the
ccloudCLI - Memory — embeds the incident's symptoms and searches CockroachDB's own native vector index for similar past incidents
- Diagnostic — if nothing similar has been resolved before, an LLM reasons over the evidence and proposes a fix (CREATE INDEX or ANALYZE); if a near-identical incident has been resolved before, the LLM is skipped entirely and the known fix is reused
- Execution — applies the fix under guardrails (confidence gating, dry-run mode, one fix per incident), re-benchmarks, and automatically rolls back index fixes that don't actually help
- Reporting — writes a natural-language incident report, cross-references CockroachDB Cloud's own audit trail, and saves the resolution back into memory for next time
The whole pipeline is visible live on a public dashboard, deployed on AWS.
How we built it
CockroachDB is the backbone, not just the thing being monitored: incident memory lives in a native VECTOR column with a CREATE VECTOR INDEX (C-SPANN), in the same cluster as the operational data — no separate vector database, no sync pipeline. We also integrated the ccloud CLI as a second CockroachDB-specific tool, pulling cluster plan/state/capacity into the diagnosis and folding the Cloud audit trail into the final report.
On the AWS side, the system is deployed on EC2 with an IAM instance role (no hardcoded credentials), running as a systemd service with a stable Elastic IP. CloudWatch and ECS handle infrastructure-aware scaling when a database-level fix alone isn't enough, and the reasoning agent can run on Bedrock as a second AWS integration point.
Challenges we ran into
Almost everything that broke only showed up when we stopped trusting mocks and tested against a real cluster. A few examples: CockroachDB silently auto-creates a secondary index on any foreign-key column that doesn't already have one — which defeated our "intentionally unindexed table" demo premise until we found it via SHOW INDEXES. Our own guardrail that added IF NOT EXISTS to replayed fixes broke again the first time the LLM proposed an unnamed index, since IF NOT EXISTS requires a name in CockroachDB's grammar. A psycopg2 parameter-substitution quirk turned a literal % in a LIKE query into a cryptic IndexError. And our dashboard was hard-failing on every request because it queried a column that never existed in our own schema — invisible until we curl'd the API directly instead of trusting the browser. Every one of these is now fixed and covered by a regression test.
Accomplishments that we're proud of
Getting the self-improving memory loop to actually work, end to end, on a real cluster: run the pipeline once, it diagnoses a genuine incident and calls the LLM; run it again, and the trace shows diagnosis (recalled) — LLM skipped — the second occurrence of a known problem costs zero model calls. We're also proud that this is a fully deployed, publicly reachable system, not a local-only demo, and that our test suite and the bugs documented in our README are honest evidence of real engineering rather than a polished-looking demo that was never stress-tested.
What we learned
That the gap between "works in a mock" and "works on a real managed cloud database" is where the real engineering lives — CockroachDB's FK auto-indexing, its grammar quirks around IF NOT EXISTS, and its managed-cluster restrictions on crdb_internal all only surfaced under real conditions. We also learned a lot about designing guardrails that fail safe: confidence gating, automatic rollback, and defense-in-depth SQL validation independent of what the LLM proposes.
What's next
Extending the incident taxonomy beyond missing indexes and stale statistics (lock contention, connection pool exhaustion), wiring Bedrock as a fully verified third reasoning backend, and exploring cross-cluster memory so a fix learned on one CockroachDB deployment can inform diagnosis on another.
Log in or sign up for Devpost to join the conversation.