Overview

The Causal Trust Agent (CTA) is an autonomous incident detection and remediation system for multi-agent pipelines. It combines real-time observability, root cause analysis, and automated patching to reduce Mean Time To Recovery (MTTR) from hours to seconds without human intervention.

Inspiration

Modern distributed systems fail in complex ways. When a multi-agent pipeline breaks in production, engineers manually sift through logs, correlate traces, hypothesize causes, and test fixes. This process typically takes 150+ seconds for simple schema drifts and hours for complex failures. We envisioned a system that could autonomously detect, diagnose, and remediate incidents while maintaining full auditability and learning from past failures.

What It Does

The CTA system provides three core capabilities:

1. Autonomous Detection The system monitors multi-agent pipeline execution through structured trace events. When anomalies occur, it captures complete execution context including agent states, tool invocations, error messages, and performance metrics.

2. Causal Analysis Using both heuristic pattern matching and optional LLM reasoning, CTA performs root cause analysis to identify the primary failure cause. It generates 5-Whys causal chains, extracts evidence from traces, and proposes specific remediation strategies.

3. Automated Remediation CTA applies schema adapters or configuration patches, runs canary tests on live data subsets, and promotes fixes to production only when success criteria are met. Failed remediations trigger automatic rollbacks.

Technical Architecture

Data Flow

The system implements a closed-loop architecture across three integrated platforms:

ClickHouse Cloud stores structured log events and audit results. The logs table contains raw pipeline execution data with fields for timestamp, component, log level, and content. The audit_results table stores anomaly detection results including latency metrics and status codes. The cta_results table persists complete remediation outcomes with before/after metrics and decision rationale.

Python Backend implements the agent pipeline and CTA logic. Trace events flow through a 3-agent graph (Intake, Retriever, Auditor) with automatic instrumentation. When failures occur, the analysis engine queries ClickHouse for relevant events, applies heuristic rules to detect schema drift patterns, and optionally invokes LLM reasoning for complex cases.

Datadog provides real-time observability dashboards. Metrics flow automatically during remediation workflows, including error rates, latency percentiles, MTTR measurements, and canary test results. The dashboard displays before/after comparisons to validate fix effectiveness.

Core Components

Trace Instrumentation Every agent step and tool invocation is captured as a structured JSONL event. Events include timing data, input/output schemas, and error details. This creates a complete execution audit trail stored in both SQLite (for fast local access) and ClickHouse (for long-term analytics).

Root Cause Analysis Engine The analyzer examines trace sequences to identify failure patterns. For schema drift, it detects field name mismatches between tool outputs and agent expectations. It constructs causal chains showing how upstream changes propagate to downstream failures. Confidence scoring ranges from 65% (pure heuristic) to 95% (LLM-enhanced).

Canary Testing Framework Before promoting fixes, CTA runs canary tests on $N=20$ production data samples. Let $E$ be the error rate and $L$ be the 95th percentile latency. A fix is promoted only if:

$$E_{canary} < 0.01 \text{ and } L_{canary} < 500\text{ms}$$

Otherwise, automatic rollback occurs.

Learning System Successful remediations are stored as incident signatures using vector embeddings. When similar failures recur, cached fixes are applied instantly without re-analysis, reducing MTTR from 45+ seconds to under 5 seconds.

How We Built It

Phase 1: ClickHouse Integration

We began by establishing bidirectional data flow with ClickHouse Cloud. The challenge was supporting both the Query API (HTTP-based) and native protocol while handling type conversions between ClickHouse's columnar format and Python dictionaries.

We implemented three key tables:

logs: Raw event data with 11 fields including LineId, Level, Component, and Content
audit_results: Evaluation results with anomaly flags, latency metrics, and HTTP status codes
cta_results: Complete remediation outcomes including analysis method, confidence scores, patch details, canary metrics, and MTTR

The integration required careful handling of timestamp formats, NULL values, and array aggregations. We used parameterized queries to prevent SQL injection and implemented connection pooling for performance.

Phase 2: Datadog Observability

Integrating Datadog required understanding StatsD metric types and proper tag structuring. We sent three categories of metrics:

Operational Metrics: cta.error_rate, cta.latency_ms with phase tags (before_fix, after_fix) to enable before/after comparison charts.

Workflow Metrics: cta.mttr_seconds, cta.workflow.duration_s broken down by analysis method (heuristic, llm, cached) to measure efficiency.

Decision Metrics: cta.canary.tests with passed/failed tags, cta.promotions.success, cta.rollbacks.count to track remediation outcomes.

The dashboard uses ordered layout with 8 widgets including timeseries for trend analysis, query values for KPIs, and bar charts for method distribution.

Phase 3: Agent Pipeline with Memory

The 3-agent graph uses LangGraph state machines. Each agent maintains isolated state:

Intake Agent: Validates input parameters against expected schemas. Uses Pydantic models for type safety. Emits validation events.

Retriever Agent: Fetches data from ClickHouse using the fetch_logs_from_cloud function. Handles pagination and connection retries. Stores results in audit_results for use as agent memory.

Auditor Agent: Evaluates each event for anomalies using business rules. Checks for schema compliance, latency thresholds, and error status codes. Uses past audit results as context.

The memory architecture allows agents to learn from previous evaluations without re-processing identical inputs.