Inspiration

AI agents are moving from demos into real customer-facing workflows, but debugging them still feels like debugging ordinary web services. Traditional logs can tell us that a model responded or a tool failed, but they rarely answer the questions that matter:

• What evidence did the agent have?
• Which observable decision changed the outcome?
• Which customers or workflows were affected?
• Was the problem isolated or part of a wider pattern?
• What should we change and replay before deploying a fix?

This becomes especially important in enterprise support. A technically “successful” agent run can still produce a frustrated customer, an unnecessary escalation, or a costly resolution.

I built Aletheia to provide operational truth for AI agents: an inspectable record of decisions, tool calls, state changes, costs, and outcomes—without collecting or claiming access to private chain-of-thought.

What it does

Aletheia is a self-hostable observability and debugging platform for OpenAI-powered agents.

It converts agent activity into structured executions containing:

• Evidence-linked decision events
• Model and function-tool calls
• MCP discovery, calls, and results
• Handoffs, retries, failures, and state changes
• Duration, token usage, and estimated model cost
• Replayable checkpoints and execution lineage
• Before-and-after execution comparisons

Instead of stopping at individual traces, Aletheia connects agent behavior to operational outcomes. Its enterprise support overview groups executions by issue, product, channel, region, sentiment, escalation, and tool health. This helps teams move from “one agent behaved badly” to “this class of customers is being affected by this specific workflow failure.”

The hosted demonstration creates ten synthetic customer-support situations and runs ten agents concurrently across three Streamable HTTP MCP servers containing twenty tools. Controlled failures make it possible to observe how degraded tools and different customer contexts affect agent decisions.

Every customer, order, ticket, refund, and escalation in the demo is synthetic. The agents and MCP interactions are real.

How I built it

Aletheia is organized around a shared, versioned execution-event protocol.

The main components are:

• A Next.js and TypeScript dashboard for authentication, project management, execution timelines, causal graphs, analytics, replay, and comparison
• Supabase Postgres, Auth, Realtime, and Row Level Security for durable ingestion and tenant isolation
• A published OpenAI Agents SDK trace processor, available as @avinash27/aletheia
• A hosted Streamable HTTP MCP server that lets Codex record executions without installing a local SDK
• The OpenAI Responses API for bounded, evidence-based trace analysis
• A standalone enterprise-support simulator using real OpenAI agent runs and MCP tools

An Agents SDK application can add Aletheia with one processor:

import { addTraceProcessor } from "@openai/agents";
import { createAletheiaTraceProcessor } from "@avinash27/aletheia";

addTraceProcessor(
  createAletheiaTraceProcessor({
    apiKey: process.env.ALETHEIA_API_KEY!,
    openaiApiKey: process.env.OPENAI_API_KEY,
    captureContent: false,
  }),
);

Content capture is disabled by default. The processor can export metadata, statuses, tool activity, timing, usage, and cost without storing sensitive model or tool content.

For Codex, Aletheia exposes MCP tools for starting and finishing traces, recording decisions and tool outcomes, inspecting executions, and processing replay requests. This project description itself was prepared while using Aletheia to observe the work.

How I used Codex and GPT-5.6

Codex was my primary development collaborator throughout the project. I used it to:

• Design the event protocol and ingestion model
• Implement the Agents SDK trace processor
• Build and debug the hosted MCP integration
• Develop the dashboard, trace explorer, and comparison views
• Write Supabase migrations and Row Level Security policies
• Create automated tests and investigate build failures
• Refine the product narrative and judge-facing demonstration

GPT-5.6 is used where semantic interpretation adds real value:

• Generating varied synthetic support scenarios
• Running the support agents that investigate those situations
• Producing concise decision summaries from observable trace evidence
• Analyzing completed executions and identifying likely root causes

The product does not ask GPT-5.6 to invent missing telemetry. Analysis is bounded by the events that were actually recorded.

Challenges I faced

Designing a useful event model

Different frameworks describe agent activity differently. Model calls, tools, MCP operations, handoffs, guardrails, retries, and custom events needed to fit one protocol without losing their original meaning.

I addressed this with a versioned envelope and a flexible event taxonomy that preserves framework-specific attributes while still supporting a consistent timeline and graph.

Preserving privacy without losing observability

Agent traces can contain customer data, prompts, tool arguments, or private identifiers. Capturing everything would make the product easier to demonstrate but unsafe as a default.

Aletheia therefore uses metadata-first capture, project-scoped API keys, tenant isolation, and opt-in content recording. It records observable evidence and never presents private chain-of-thought as telemetry.

Handling concurrent executions correctly

The enterprise demo runs ten agents simultaneously. API keys, traces, batches, events, and metrics must remain associated with the correct execution even when events arrive concurrently or out of order.

I built buffered delivery, project-scoped routing, idempotent ingestion, execution rollups, and shared batch identifiers to keep these runs isolated and comparable.

Turning traces into operational insight

A timeline alone is useful to developers, but it does not immediately tell an operations team whether a problem affects one customer or an entire region.

The support overview was designed to calculate metrics from real execution metadata and tool outcomes, connecting individual failures to issue clusters, escalation rates, customer sentiment, operational hotspots, and tool reliability.

Making replay honest and reproducible

Replay is more than pressing “run again.” The new execution must preserve its relationship to the original run and identify the checkpoint and context that changed.

Aletheia stores replay requests, source execution lineage, context patches, and child execution IDs so teams can compare the original behavior with the attempted fix.

What I learned

The biggest lesson was that agent observability cannot end at model input and output. An agent is a workflow involving models, tools, state, policies, external systems, and business outcomes. Those elements need to be visible in the same execution.

I also learned that technically successful agent runs can still represent product failures. A 200 OK does not mean the customer received the correct outcome.

Finally, I learned that replay becomes much more valuable when it is connected to measurable results. The important question is not only “did the second run behave differently?” but also whether it reduced escalation, improved resolution, lowered latency, or used fewer resources.

For example, teams could track:

[

\text{Cost per resolved case}

\frac{\text{Total model and tool cost}} {\text{Number of successfully resolved cases}} ]

What I’m proud of

Aletheia is not a dashboard over static mock data. It includes a published SDK integration, a hosted MCP server, durable authenticated ingestion, production deployments, real concurrent agent runs, automated tests, and a judge-ready hosted demonstration.

Most importantly, it creates a short path from:

“The agent behaved badly.”

to:

“We know which observable decision or tool changed the outcome, who was affected, and what to replay.”

What’s next

The next steps are broader framework adapters, richer evaluation datasets, team collaboration, alerting, configurable operational metrics, and automated regression suites built from production failures.

The long-term goal is to make Aletheia a shared operational layer where agent developers, AI-platform teams, and business operators can understand autonomous systems using the same evidence.

Built With

• codex
• nextjs