Delta Agent Observability

Agents run. Delta watches.

Comment

Inspiration

Everyone ships AI agents. But once they're running, they become a black box. No way to see how they're doing, no way to understand why they're producing bad outputs, and no way to help them get back on track. Delta came from this frustration: there's nothing built for the way AI agents fail.

What it does

Delta is an observability platform for AI agents. It tracks every run, capturing tool calls, LLM calls, retries, token usage, and latency. Across runs, it computes:

  • Drift Detection: cosine similarity between each run's behavioral fingerprint and the agent's rolling baseline, flagging when behavior meaningfully shifts
  • Anomaly scoring: z-score baselines per metric (duration, tokens, tool calls, error rate) to surface statistical outliers
  • Fatigue tracking: linear regression to detect gradual degradation trends
  • Process conformance: Markov chain model over tool call sequences to catch agents going off-script
  • Severity tiers: weighted aggregation into normal / watch / warning / alert / critical

All of this is pushed to a real-time dashboard with run history by agent, signal layer activation status, metric charts, and a trace timeline for every run.

How we built it

  • Backend: FastAPI & PostgreSQL. Event ingestion API receives events and an assembler polls to assemble complete runs and compute all analysis scores
  • SDK: Lightweight Python client with a context manager interface that instruments any agent loop
  • Analysis Pipeline: Python: cosine similarity, Laplace-smoothed Markov chains, scipy linear regression
  • Dashboard: Next.js
  • Test Agents: Claude-powered code agent and web search agent

Challenges we ran into

  • Z-score drift was meaningless. Our first approach normalized feature vectors using z-scores, but typical runs all produce near-zero vectors, making cosine similarity return 0 and drift score always 1.00 for every normal run. We rethought the math and switched to raw-normalized values (divided by scale factors), which gives each run a unique behavioral fingerprint.
  • Agent flush never called. The coding agent was reporting timed_out on every run. Root cause: obs.flush() was after a return inside the with block, daemon threads don't guarantee execution at process exit. Fixed with try/finally and break instead of return.
  • Rate limits on the free tier. Long agent conversations hit 10k tokens/min. Added retry logic with 60s × attempt exponential backoff.

Accomplishments that we're proud of

  • A fully working end-to-end observability pipeline: from SDK instrumentation through event ingestion, assembly, multi-layer analysis, and live dashboard
  • The drift detection math actually works. After fixing the z-score bug, real runs show drift scores of 0.01–0.15 for typical behavior and 0.3–1.9 for outliers (timed-out, rate-limited runs).
  • Two real instrumented agents generating genuine observability data, not mock data.

What we learned

  • Observability for agents is fundamentally different from observability for APIs. The unit of analysis isn't a request, it's a behavioral session with tool use, multi-step reasoning, and emergent patterns across runs.
  • Cosine similarity on feature vectors is a surprisingly effective behavioral fingerprint, but the normalization strategy matters more than the algorithm.
  • The hardest part of building developer tooling isn't the features, it's making instrumentation so lightweight that no one has an excuse not to use it.

What's next for Delta Agent Observability

  • Multi-agent support: trace spans across orchestrator + subagent hierarchies
  • Alerting: webhook and Slack notifications when severity crosses a threshold
  • Prompt drift detection: track when system prompt changes correlate with behavioral shifts
  • Cost tracking: per-run and per-agent token cost attribution with budget alerts
  • Agent comparison: side-by-side behavioral diff between two agents or two time windows
  • OpenTelemetry export: pipe Delta signals into existing observability stacks

Built With

Share this project:

Updates

Submission history