Inspiration
AI teams have strong tools for collecting traces, prompts, latency, and token usage, but a failed agent run still leaves a developer manually searching events and guessing at the cause. We wanted to move beyond “here is the trace” to “here is the failure, the evidence, the fix, and a way to verify it.”
AgentOps Studio explores what debugging looks like when GPT-5.6 is the reasoning engine while deterministic software remains responsible for verification.
What it does
AgentOps Studio turns a failed AI-agent execution trace into a complete debugging workflow. In our synthetic Build Week scenario, a customer-support refund agent finds a customer ID, sends customer_name to a tool that requires customer_id, retries the unchanged invalid call, and fails.
GPT-5.6 analyzes the normalized trace and returns a structured diagnosis with:
- a root cause and contributing factors;
- observed facts separated from inferences;
- evidence strength and missing telemetry;
- a recommended developer-controlled action; and
- citations to exact trace-span IDs.
A developer can jump from each claim to its supporting evidence, generate and review a declarative regression evaluation, and run that evaluation deterministically against the failed run. The developer then explicitly selects a prepared agent version and replays the scenario in a mocked sandbox. The same reviewed evaluation passes, and the UI calculates the before-and-after changes in latency, cost, tokens, spans, and tool calls.
When telemetry is incomplete, AgentOps Studio returns an explicit insufficient-evidence result rather than inventing a root cause. If live model access is unavailable, the demonstration uses an honestly labeled, schema-validated cached result previously generated by the same GPT-5.6 workflow.
How we built it
AgentOps Studio is a single Next.js 16 App Router application written in strict TypeScript and React 19. Zod defines bounded contracts for normalized traces, diagnoses, and declarative evaluations. Synthetic fixtures provide deterministic broken, retry, fixed, and insufficient-evidence runs.
The OpenAI TypeScript SDK connects to GPT-5.6 through a LiteLLM OpenAI-compatible Responses API endpoint with strict structured output. The application validates every cited span and rejects evaluation predicates outside a closed vocabulary. GPT-5.6 interprets evidence and drafts tests; TypeScript code owns pass/fail and every comparison metric.
Reviewed evaluations, normalized runs, replay lineage, and results are persisted in DuckDB. Replay is deliberately constrained to one synthetic refund-agent workflow, with mocked tools that can record intent but cannot contact refund, email, payment, or customer systems.
Vitest covers schemas, citation integrity, deterministic evaluation, replay safety, sanitization, configuration, and persistence. Playwright exercises the complete credential-free golden path. A standalone Debian-based Docker image provides a reproducible deployment target.
How ChatGPT and Codex accelerated the build
Before implementation, I used ChatGPT as a product-thinking partner to compare several possible Build Week directions: a broad agent platform, observability dashboards, evaluation tooling, replay, governance, and knowledge graphs. That brainstorming exposed the stronger product wedge: do not merely display a trace; explain the failure from evidence and prove whether a fix works. This narrowed the idea into the evidence-to-proof workflow implemented in AgentOps Studio.
I then used two Codex threads with distinct responsibilities:
- Core developer thread (
019f6216-1495-7071-a43f-8c1753bd8fa6): planning, architecture, implementation, tests, persistence, safety, and release work. - Reviewer thread (
019f6c2c-cfdb-7242-a672-e64a4a5da5e1): independent review of the plans and source code, evidence and claim checking, risk discovery, and demo critique.
The two threads communicated asynchronously through repository-local Markdown. The core thread wrote the brainstorm, product plan, backlog, ADRs, typed contracts, implementation notes, and verification evidence. The reviewer read those artifacts alongside the code and returned structured critiques in review.md, review_v2.md, and review-demo-story-line.md. The core thread then reconciled the findings into TASKS.md, the implementation, tests, and final demo. Markdown acted as a shared protocol and left an auditable path from idea, through code review, to revision.
Across that loop, Codex helped:
- turn the initial product thesis into a bounded vertical slice and dependency-aware backlog;
- document eight architecture decisions before implementation;
- scaffold the strict Next.js, TypeScript, ESLint, Prettier, Vitest, and Playwright environment;
- implement the trace schema, GPT-5.6 diagnosis boundary, closed evaluation DSL, deterministic evaluator, sandboxed replay, DuckDB adapter, and product UI;
- catch unsafe partial LiteLLM configuration and replace it with atomic key-and-endpoint selection;
- add trace redaction, truncation, cached fallback categories, spend controls, reset protection, and container packaging; and
- run iterative lint, type, unit, production-build, live-model, browser, and container verification loops.
The reviewer loop drove a key product boundary: GPT may interpret and propose, but deterministic application code must verify. The detailed collaboration record is preserved in the repository’s Codex build log.
Challenges we ran into
The hardest challenge was preventing a convincing explanation from being mistaken for a correct one. We separated observed facts from inferences, required facts to cite real span IDs, and reject model output containing unknown citations. Missing telemetry became a first-class insufficient-evidence state.
A second challenge was deciding where AI should stop. GPT-5.6 is excellent at interpreting a trace and proposing a regression test, but it should not decide whether its own test passes. We created a small declarative evaluation language and a deterministic evaluator.
Finally, safe replay can easily become a generic orchestration platform. For Build Week we deliberately constrained replay to one synthetic workflow with mocked side effects. That made the end-to-end result testable, reproducible, and honest.
Accomplishments that we're proud of
- Diagnosis claims focus the exact supporting spans in the trace.
- Unsupported citations and evaluation predicates are rejected at the application boundary.
- Missing telemetry produces
insufficient_evidenceinstead of a guessed root cause. - One reviewed evaluation fails the broken run and passes the corrected sandbox replay.
- Before-and-after metrics are calculated from stored run data rather than supplied by the model.
- The complete workflow is deterministic, credential-optional, and demonstrated in 2 minutes 10 seconds.
- The trace, cited diagnosis, reviewed evaluation, replay lineage, and verified result become reusable run-level debugging knowledge.
What we learned
AI-native developer tools need stronger boundaries, not fewer. Structured output is only the start: model claims must be checked against domain evidence, and deterministic code should retain authority over verification and metrics.
We also learned that a narrow, polished workflow communicates more value than a broad observability dashboard. Treating fixtures as stable product contracts let us iterate quickly without losing trust in the demo.
What's next
Today, knowledge is scoped to persisted runs and their replay lineage. Next, we want to add a bounded OpenTelemetry ingestion adapter and a cross-run context graph connecting runs, agent versions, prompts, tools, evaluations, and verified fixes. That would let teams ask: Have we seen this failure before? Which change caused it? Which fix actually held?
After that: instrumentation adapters, prompt and configuration diffs, saved-evaluation CI checks, and additional agent-specific replay vocabularies. Generic replay and production integrations will come only after their safety boundaries are proven.
Repository and testing
Source code, MIT license, setup instructions, synthetic sample data, supported-platform notes, and a credential-free judge path are available at:
Built With
- codex
- docker
- duckdb
- gpt-5.6
- litellm
- next.js
- openai-responses-api
- playwright
- react
- typescript
- vitest
- zod