TraceLens — AI Agent Debugger

Debug AI agents like you debug code. TraceLens captures every step of an agent run, flags failures (loops, bad tool calls, timeouts), and uses GPT-5.6 to explain what went wrong.

Comment

Inspiration

When AI agents fail, it's hard to see why. A retry loop silently burns time. A tool call errors out three steps deep. A step times out with no clear signal. As someone actively building agentic systems, I kept running into this exact problem logs alone don't tell you what happened in a run, they just tell you something happened. I wanted a fast, visual way to see the full picture: every step an agent took, what failed, and why.

What it does

TraceLens is an observability and debugging tool for AI agent workflows. It:

  • Ingests every step of an agent run through a simple event schema (agent_id, step_id, step_type, input, output, status, latency, parent_step_id)
  • Automatically detects three common failure patterns: retry loops, tool-call errors, and silent timeouts
  • Explains failures in plain English using an LLM, strictly grounded in the actual trace data no speculation beyond what's recorded
  • Replays the run visually in a browser-based timeline, with flagged steps clearly marked and expandable to show the explanation and suggested next debugging action

It ships with three demo scenarios including a mixed-failure customer-support run distinct from the original research-agent examples to prove the detection logic generalizes rather than being hardcoded to one example.

How we built it

I built TraceLens entirely in the OpenAI Codex desktop app using GPT-5.6 (Terra model), working in scoped, spec-driven sessions across roughly two days:

  1. Schema and storage — defined the trace event schema first, then had Codex build the FastAPI ingestion/retrieval API with a SQLAlchemy/SQLite storage layer
  2. Failure detection — three detectors (retry-loop, tool-call error, silent timeout) plus a toy agent generating realistic failure data, verified with real unit tests
  3. AI explanations — a provider-agnostic Explainer interface, with a Groq-backed implementation and an automatic offline rule-based fallback so the app never breaks if no API key is configured
  4. Replay UI — a dark-themed, dependency-free HTML/JS timeline served directly by FastAPI, no build step required
  5. Generalization testing — a third, structurally different demo scenario to prove the system wasn't just overfit to its first two examples
  6. Deployment — deployed live on Render with auto-seeding demo data on startup, so judges can test it instantly with zero setup

Every piece was written by Codex from a detailed prompt describing the exact schema, endpoints, and behavior needed, and in most cases Codex self-tested its own output using FastAPI's test client or Python's unittest before handing it back.

Challenges we ran into

The most instructive one: after wiring up Groq for live explanations, calls kept failing with an authentication error even after generating a fresh API key. Debugging alongside Codex, I traced it to a subtle interaction between python-dotenv and pre-existing shell environment variables — a stale key from an earlier terminal session was silently overriding the .env file's value. I had Codex add explicit .env loading to the app's startup, which fixed it for good.

What stood out during this bug: the app never crashed. It gracefully fell back to the rule-based explainer the entire time the Groq integration was broken validating that the resilience design (a swappable explainer interface with an offline default) worked exactly as intended, even under a real failure.

A second challenge was a bloated requirements.txt accidentally generated from a global pip freeze rather than the project's actual dependencies including a broken local editable-install path that would have crashed the Render deployment outright. Catching and fixing that before deploying was a good reminder that generated output still needs a careful human review pass, not blind trust.

Accomplishments that we're proud of

  • A fully working, end-to-end pipeline ingestion, detection, explanation, and replay built and verified in under two days
  • Genuine generalization testing: instead of stopping at one working demo, we deliberately built a third, structurally different scenario (a support-triage agent, not another research agent) to prove the detectors and UI aren't overfit
  • A resilient architecture: the explanation layer degrades gracefully to an offline rule-based fallback if the LLM provider fails or no API key is set the app is never fully broken
  • A live, publicly testable deployment on Render with automatic demo-data seeding, so anyone can try it instantly with zero setup
  • Every claim was independently verified manually testing each endpoint via /docs and each UI interaction in the browser, not just trusting Codex's generated summaries

What we learned

Working with Codex on a project like this reinforced that the highest-leverage work wasn't writing code line-by-line it was writing precise specs, verifying every claim Codex made, and making the handful of architecture calls that mattered: keeping the explainer swappable, deliberately testing generalization, and scoping the project tightly enough to actually finish and deploy it within the hackathon window.

What's next for TraceLens — AI Agent Debugger

  • Persistent storage (Postgres) for production use beyond the demo
  • More failure detectors (e.g. hallucinated tool calls, budget/cost overruns)
  • Support for streaming trace ingestion from live agent runs, not just post-hoc replay
  • Integration hooks for popular agent frameworks (LangGraph, CrewAI) to auto-instrument traces without manual event posting

Built With

Share this project:

Updates