GhostOps

Production incidents are expensive, stressful, and worst of all repetitive. Over 60% of enterprise outages are recurrences of previously solved issues. When engineers leave, switch teams, or forget the subtleties of a 3:00 AM fix, critical operational context evaporates.

Standard AI "Copilots" make the problem worse: naive RAG retrieves outdated runbooks and executes destructive scripts without understanding that the cluster topology, kernel version, or database engine has changed.

We asked: What if production infrastructure had an immutable, governed memory that survives team turnover, questions its own assumptions, and never makes the same mistake twice? That vision is GhostOps.

What GhostOps Does GhostOps is an autonomous infrastructure intelligence and governed remediation system. It transforms unstructured incident postmortems into a living, vector-indexed system of record with continuous drift detection and multi-phase safety verification.

The end-to-end closed loop: $$\text{Observe} \longrightarrow \text{Investigate} \longrightarrow \text{Retrieve Memory} \longrightarrow \text{Diff Environment} \longrightarrow \text{Govern & Plan} \longrightarrow \text{2PC Saga Execute} \longrightarrow \text{Independent Verify} \longrightarrow \text{Learn & CDC Stream}$$

Key Capabilities: CockroachDB Institutional Memory Vault: Stores incident root causes, reasoning trees, and execution traces using native VECTOR(1536) indexes. Negative Knowledge & Supersession: Indexes failed historical fixes ("DO NOT REPEAT") so the system never re-attempts hazardous actions. 9-Dimensional Temporal Drift Chamber: Evaluates 9 physical layers (Runtime, Topology, Config, Region, Dependencies, Scale, Traffic, State, Database Engine) before approving any precedent. Two-Phase Commit (2PC) Saga Engine: Idempotency locks, pre-execution snapshots, and automated reverse compensating rollbacks. Decoupled AWS Verification Reader: Never trusts execution output; independently polls AWS CloudWatch and EC2 metrics to calculate true recovery deltas. Change Data Capture (CDC) Streams: CockroachDB changefeeds stream validated memory updates to distributed regional sentinels. Dedicated 11-Act Judge Mode: A 2-minute interactive cinematic theater in Next.js 14 / Three.js demonstrating the full architecture in real time. How We Built It GhostOps was engineered as a high-reliability, production-grade monorepo across 10 structured architectural stages:

Database & Memory: CockroachDB Cloud Serverless with 1536-dimensional native vector search (pgvector cosine similarity + relational ACID filters) and Change Data Capture (CDC). AI & Multi-Agent Orchestration: Amazon Bedrock Mantle utilizing a multi-tier LLM hierarchy: Fast Tier: zai.glm-4.7-flash for sub-200ms telemetry hashing and anomaly triage. Reasoning Tier: deepseek.v3.2 for multi-hypothesis root cause trees and causal inference. Embedding Tier: amazon.titan-embed-text-v2:0 for 1536-dim semantic state signatures. Agent Framework: LangGraph multi-agent state machines. Backend: FastAPI (Python 3.12), SQLAlchemy 2.x async ORM, Alembic migrations, Pydantic v2 schemas, Redis distributed locks, and structured JSON logging. Frontend: Next.js 14 (App Router), TypeScript, TailwindCSS, Lucide Icons, and Three.js for the Obsidian Vault neural 3D cosmos. Cloud Infrastructure: Deployed on AWS EC2 (t3.small) in eu-north-1 running Docker Compose multi-container services with Nginx reverse proxy.

Benchmark & Evaluation Results We evaluated GhostOps against a curated 30-case golden benchmark corpus across Development, Validation, and Holdout splits:

Metric GhostOps Engine Traditional RAG / Runbooks Precision @ 1 (Top-1 Accuracy) 93.33% 36.67% Precision @ 3 (Top-3 Coverage) 100.00% 60.00% Mean Reciprocal Rank (MRR) 0.9667 0.4611 Temporal Drift Detection 100.00% 0.00% (Blind Replay) Unsafe Replay Violations 0.00% (Zero) 43.33% Destructive Replay Backend Test Suite 74 Passed (100%) 0 Skipped

Challenges We Conquered Preventing Destructive Historical Replay: An old fix that worked on PostgreSQL 13 will corrupt a cluster on CockroachDB 23.2. We solved this by creating the 9-Dimensional Temporal Drift Chamber, which detects schema and engine deltas and generates fresh, drift-aware action plans. Decoupled Verification vs Execution: A script returning exit code 0 does not mean the system recovered. We decoupled the execution engine from verification, writing an isolated AWS telemetry reader that computes post-remediation metric deltas over a 60-second window. Deterministic Safety Enforcement: We instituted a strict boundary: "LLMs recommend. Code governs. Humans authorize. Systems execute." High-risk actions cannot bypass deterministic safety gates regardless of LLM confidence.

Accomplishments We're Proud Of Built a complete, closed-loop autonomous operational learning cycle from telemetry anomaly to vector consolidation. Achieved 0.00% Unsafe Replay Rate and 93.33% Precision@1 on our golden benchmark. Implemented Negative Knowledge indexing, ensuring failed remediations actively protect future runs. Designed an Obsidian Vault 3D neural memory interface with an interactive 11-Act Judge Mode. Maintained 74 passing unit & integration tests with 100% test coverage across all 10 architectural stages. Fully deployed and live on AWS EC2, CockroachDB Cloud, and Amazon Bedrock.

What's Next for GhostOps Multi-Cloud Mesh Sentinels: Extending autonomous changefeed sentinels across AWS, GCP, and Azure. Causal DAG Incident Forecasting: Pre-incident anomaly trajectory projection to remediate resource exhaustion before alarms fire. Autonomous Synthetic Chaos Replays: Nightly automated dry-run simulations in isolated staging sandboxes to validate institutional memory validity against daily CI/CD deployments.