TraceLog — Continuous Reliability for AI Agents

A GPT-5.6 meta-agent that catches production-agent failures, creates regression assets, and proves safer prompt fixes before human approval.

Comment

Inspiration

Production AI agents can return fluent, confident failures that uptime monitoring never sees. A single hallucinated policy or broken tool call can reach customers before an engineer notices. TraceLog is the reliability engineer that watches those traces, explains the failure, and proves a safer fix.

What it does

TraceLog supervises an AI agent through Arize Phoenix and turns one incident into a complete, auditable repair loop:

  1. Capture and deduplicate the failing trace.
  2. Diagnose the failure with confidence and severity.
  3. Build an evidence-backed causal chain.
  4. Produce a typed remediation plan.
  5. Generate adversarial regression cases.
  6. Evaluate the baseline and candidate.
  7. Prepare a versioned prompt patch.
  8. Replay the original incident.
  9. Red-team the candidate with unseen, embedding-filtered holdouts.

Prompt candidates are evaluated but never silently promoted. Code, tool, data, and configuration remediations require human approval.

How we built it

The Python/FastAPI services use the OpenAI Responses API with structured Pydantic outputs. GPT-5.6 Sol performs diagnosis, root-cause analysis, remediation, synthesis, and patch generation. GPT-5.6 Terra handles repeated evaluation and Patient-agent calls. OpenAI embeddings enforce semantic separation between generated development cases and final holdouts.

Phoenix supplies trace and experiment observability. A React, TypeScript, Vite, and SSE cockpit exposes every stage, artifact, score, and lineage record. TraceLog also publishes an MCP server and a CI prompt gate for developer workflows.

How Codex helped

Codex was the primary implementation partner. It analyzed the original architecture, migrated the model layer to GPT-5.6, implemented strict Responses function tools, hardened authentication and deployment boundaries, added semantic holdout filtering and lineage, wrote unit, integration, and Playwright tests, reviewed dependency updates, created the judge-safe fixture, and shipped the GitHub Pages deployment. Key decisions were verified through protected pull requests and CI rather than accepted without review.

Challenges and lessons

The largest challenges were preventing test traffic from recursively triggering supervision, keeping holdouts truly unseen, and separating fixture evidence from live model evidence. Phoenix MCP currently lacks the required annotation write tool, so TraceLog uses its official REST API as a narrow fallback.

Because funded API validation was unavailable, the public judge experience is a clearly labelled deterministic offline fixture. It makes no OpenAI, Phoenix, or Patient calls and is not presented as live GPT-5.6 evidence. The real integration, tests, setup instructions, and credential-dependent live path are all available in the repository.

Try it

Open the public demo and click Play offline fixture, then Show fixture scorecard. No account, backend, API key, or billing is required.

Built With

Share this project:

Updates