CortexOPS: AI agent for CockroachDB with memory.

CortexOps is an AI agent for CockroachDB that autonomously diagnoses incidents using AWS Bedrock, executes gated fixes, and stores decision rationale in distributed vector memory for instant recall.

Comment

Inspiration

Operating distributed SQL databases like CockroachDB at scale is demanding. During 3 AM outages or sudden traffic spikes, site reliability engineers (SREs) and database administrators must rapidly diagnose telemetry, identify transaction contention, and execute remediation actions.

However, modern database operations suffer from a critical flaw: Operational Amnesia. When an engineer scales a cluster, drops an index, or adjusts zone configurations during an incident, the reasoning rationale behind that action disappears into Slack channels or mental memory. Months later, no one can answer: "Why did we make this decision?"

We built CortexOps to bridge the gap between autonomous AI-driven infrastructure self-healing and long-term auditable operational memory.

What it does

CortexOps is an autonomous, AI-assisted operations agent that continuously observes CockroachDB telemetry, diagnoses anomalies, executes safety-gated remediation, and indexes operational memory:

  1. Continuous Telemetry Observe Loop: Monitors live CPU utilization, active query count, transaction contention locks, and Raft replication quorum.
  2. AI Reasoning Grounded in Domain Skills: When anomalies occur (e.g. CPU $> 80\%$ or lock contention), CortexOps consults verified CockroachDB Agent Skills (performance-and-scaling, query-and-schema-design, operations-and-lifecycle) and uses Amazon Bedrock (Claude 3.5 Sonnet) to synthesize human-readable remediation proposals.
  3. Human-in-the-Loop Safety Gate: Operators maintain full control. Remediations are verified against an immutable prefix whitelist (ccloud cluster scale, ccloud cluster backup) before execution.
  4. Persistent Vector Semantic Memory: Every decision's reasoning text is converted into a 1024-dimensional vector embedding and stored directly inside CockroachDB.
  5. Natural Language Semantic Recall (cortexops ask): Operators can query cluster memory in plain English (cortexops ask "why did you scale up the cluster?") to perform real-time vector similarity search over historical operational rationale.

How we built it

  • CockroachDB Distributed Vector Store: Utilized CockroachDB's native VECTOR(1024) datatype and distributed C-SPANN Cosine Distance Indexing (embedding vector_cosine_ops) to store operational decisions alongside telemetry. Cosine similarity is computed directly on database engine nodes: $$\text{Distance}(\vec{u}, \vec{v}) = 1 - \frac{\vec{u} \cdot \vec{v}}{|\vec{u}|2 |\vec{v}|_2} = 1 - \frac{\sum{i=1}^{1024} u_i v_i}{\sqrt{\sum_{i=1}^{1024} u_i^2} \sqrt{\sum_{i=1}^{1024} v_i^2}}$$
  • Amazon Bedrock AI Engine: Integrated anthropic.claude-3-5-sonnet for structured reasoning proposals and cohere.embed-english-v3 for 1024-dim vector embedding generation.
  • Go Systems CLI (cortexops): Built in Go 1.25 with Cobra, Charmbracelet Bubbletea, Huh, and Lipgloss. Features a smart entry point that automatically triggers cluster onboarding when disconnected, and renders an interactive dashboard when connected.
  • Live Operator Web Dashboard: Built in React 19 + Vite, featuring real-time time-series telemetry charts (Recharts), animated decision feeds, in-browser cluster connection wizards, and semantic memory search.
  • CockroachDB Agent Skills: Embedded specialized YAML runbooks to deterministically guide the LLM according to CockroachDB architectural best practices.

Challenges we ran into

  1. Distributed Vector Dimension Alignment ($d=1024$): Aligning Cohere embedding output vectors with CockroachDB's VECTOR(1024) C-SPANN index required precise float serialization and zero-copy string formatting between Go, Node.js, and CockroachDB SQL drivers.
  2. Preventing Autonomous Hallucinations: Autonomous agents can be dangerous if unconstrained. We solved this by enforcing two guardrails:
    • Dynamic prompt injection of deterministic CockroachDB Agent Skills.
    • An independent Go Safety Gate (:5005) that validates CLI command prefixes against an immutable whitelist before execution.
  3. Zero-Crash Offline Fallback: Ensuring that developers without active AWS Bedrock credentials can still run the entire system smoothly using pre-synthesized CockroachDB operational rule matching.

Accomplishments that we're proud of

  • Native CockroachDB Vector Storage: Demonstrating that CockroachDB is not only an exceptional relational database, but also a first-class Vector Database capable of powering AI memory search.
  • Smart CLI Entry Point: Seamless terminal experience where running cortexops auto-detects cluster health in under 500ms and displays the appropriate onboarding or operational interface.
  • Human-in-the-Loop SRE Workflow: Giving operators confidence by pairing AI-generated reasoning with explicit approval controls and persistent audit logging.

What we learned

  • How to integrate Model Context Protocol (MCP) and CockroachDB Agent Skills into foundation models to ground LLM reasoning in verified database operational runbooks.
  • The mechanics of CockroachDB's C-SPANN vector index and how vector cosine distance calculations scale across distributed nodes.
  • Crafting rich, accessible terminal user interfaces (TUIs) using Go and Charmbracelet libraries.

What's next for CortexOPS: AI agent for CockroachDB with memory.

  • Predictive Anomaly Forecasting: Machine learning time-series models to predict storage growth and CPU bottlenecks hours before they trigger alerts.
  • Automated Multi-Region Rebalancing: Extending Agent Skills to autonomously suggest REGIONAL BY ROW and geo-partitioning optimizations based on live latency telemetry.
  • Slack & PagerDuty Gated Approvals: Native webhook integrations allowing engineers to review AI decision reasoning and approve ccloud scale actions directly from Slack.

Built With

  • anthropic-claude
  • aws-bedrock
  • ccloud
  • charmbracelet-bubbletea
  • cloudwatch
  • cobra-cli
  • cockroachdb
  • cohere
  • express.js
  • go
  • lambda
  • lipgloss
  • node.js
  • react
  • recharts
  • vector-search
  • vite
Share this project:

Updates

Submission history