Synapse Ops Guard

An evidence-governed incident response and decision guardrail engine for autonomous operations.

Comment

Inspiration

The rise of autonomous operational AI agents promises to dramatically reduce Mean Time to Resolution (MTTR) by diagnosing and fixing production incidents automatically. However, deploying unchecked AI agents in live environments poses massive risks—such as initiating destructive rollbacks based on weak correlations (hallucinations), mutating service configurations silently without an audit trail, or triggering state race conditions. We were inspired to build Synapse Ops Guard to bridge the trust gap between automated agentic logic and actual production execution. We wanted to prove that safe automation is not just about acting faster; it is also about knowing when not to act.

What it does

Synapse Ops Guard acts as an evidence-governed decision control layer and state machine for operational AI agents. It maps simulated Splunk-style telemetry (alerts, metrics, traces, and deployment logs) to a deterministic policy core.

  • Scenario A (Approval-Gated Rollback): It detects a high-severity regression, correlates it with a recent deployment, recommends a rollback, gates the execution on manual operator approval, simulates a safe dry-run execution (DEMO_SAFE), verifies the system recovery against baseline metrics, and generates a chronological audit timeline of the event.
  • Scenario B (Formal Abstention): When telemetry is ambiguous or conflicting, the system formally abstains from taking action and closes the session safely, preventing compounding incidents.
  • Optional Splunk HEC Export: It includes a pathway to export compiled audit timelines directly to a real Splunk index via the HTTP Event Collector.

How we built it

We structured Synapse Ops Guard as a TypeScript monorepo managed with pnpm workspaces and turbo pipeline caching.

  • Backend API: Built using NestJS and Node.js, managing session state machines, snapshots, and API controllers.
  • Frontend Dashboard: Built using Next.js 16, React 19, and Tailwind CSS, exposing a beautiful dark-mode interface with stepper indicators showing the live governance transition.
  • Database & ORM: PostgreSQL database backed by Prisma ORM for tracking persistent state.
  • Decision Engine: Formulated as a decoupled TypeScript library (@synapse/decision-core) running deterministic rule verification.
  • Automation & Reset Scripts: PowerShell automation scripts (pnpm demo:a, pnpm demo:b, etc.) to run full end-to-end telemetry scenarios directly from the command line.

We model the risk boundary mathematically: An action $A$ is recommended with confidence $C(A)$ based on evidence parameters. If the correlation confidence $C(A)$ falls below a strict threshold $\theta$, or if the telemetry contains conflicting signals, the system outputs an abstention decision: $$\text{Decision} = \begin{cases} \text{Recommend}(A) & \text{if } C(A) \ge \theta \ \text{Abstain} & \text{if } C(A) < \theta \end{cases}$$

Challenges we ran into

  • Monorepo Linting and Typing Orchestration: Aligning modern ESLint v9 flat configurations and TypeScript compiler options (rootDir settings) across shared workspace packages (contracts and decision-core) and applications (api and web).
  • Designing Formal Abstention Logic: Ensuring that "doing nothing" is treated as an active, first-class state-machine transition in the database, bypasses execution, and writes a detailed safety justification to the audit logs.
  • Simulating Telemetry Safely: Crafting realistic Splunk-style mock records (representing alerts, trace packet losses, and latency baselines) so that the entire flow runs predictably and safely in sandbox environments without external dependencies.

Accomplishments that we're proud of

  • Robust State-Machine Gating: Enforcing database-level state validation rules so that no execution can happen out-of-order or without a logged operator approval record.
  • Fully Interactive Stepper Dashboard: A premium, live-updating UI that shows exactly which gate (Evidence, Decision, Approval, Execution, Verification, Closure) the incident response is currently passing through.
  • Zero-Compile-Error Monorepo: Creating a cohesive code structure where the API, Web console, decision library, and schema contracts are fully typed and build with 100% test and lint validation.

What we learned

  • AI-assisted operations cannot be treated as simple chat prompts or direct command runners. They require a deterministic governance layer to prevent natural language drift.
  • Building robust developer and judge workflows with single-command bootstrap scripts (pnpm dev, pnpm install) and database reset automation dramatically increases the ease of evaluation and deployment confidence.

What's next for Synapse Ops Guard

  1. Production Cluster Adapters: Building real Kubernetes rollback controllers and LaunchDarkly feature-flag adapters to replace dry-runs.
  2. Splunk Webhook Ingest: Moving from mock telemetry to live Splunk Cloud webhook receivers for metric and alert ingestion.
  3. Operator Identity Verification: Adding OIDC authentication and cryptographic signatures to operator approval records.

Built With

Share this project:

Updates