Trace Doctor

Inspiration

About TraceDoctor

TraceDoctor is an AI debugging copilot for agent developers. It turns failed agent traces into a root cause, trace-backed evidence, a targeted fix, and a regression test.

The inspiration came from a common agent development problem: tracing tools can show what happened, but developers still have to manually figure out why the agent failed and what to change. If an agent calls hotel_search when the user asked for a flight, the trace makes the mistake visible, but the developer still has to identify the failure category, locate the first bad step, decide whether to fix the prompt or tool description, and create a test so the bug does not come back.

What it does

TraceDoctor focuses on that missing layer: diagnosis.

The workflow is:

Failed trace -> Root cause -> Evidence -> Recommended fix -> Regression test

For example, given a trace where the user asks for a flight but the agent calls a hotel tool, TraceDoctor outputs: - Failure category: tool selection - Failed step: step 1 - Root cause: the agent selected hotel_search for a flight request - Evidence: user asked for a flight, but the first tool call was hotel_search - Recommended fix: clarify the tool description - Regression test: expect flight_search, forbid hotel_search

I built the project as a focused MVP rather than a full observability platform. The goal was not to replace tracing, but to sit on top of traces and help developers move from observing failures to fixing them.

Challenges we ran into

The hardest challenge was scope. It would be easy to accidentally build a full observability dashboard, but that would be too broad for a hackathon MVP. I narrowed the product to one core workflow: I can see what happened. Now tell me why it failed and what to change. Another challenge was making the result feel more useful than a generic LLM judge. To address this, TraceDoctor uses: - A fixed failure taxonomy - Step-level failed-step localization - Trace-backed evidence - Explicit fix targets - Structured regression test output This makes the output more actionable and easier to verify.

Accomplishments that we're proud of

What we learned

I learned that agent debugging is most useful when it is step-level and actionable. A generic judge can say "the answer is wrong," but that is not enough for developers. The more useful question is: Where did the agent first go wrong, and what should I change?

I also learned that regression tests are a powerful bridge between debugging and evaluation. Once a failure is diagnosed, it should become an eval case so the same bug does not return later. In that sense, TraceDoctor is not just a trace analyzer. It is a failure-to-eval compiler.

How we built it

TraceDoctor is built as a web app with a structured diagnosis pipeline. The app accepts a simplified agent trace JSON format:

{
  "user_request": "Book me a flight to Tokyo",
  "available_tools": [
    {
      "name": "flight_search",
      "description": "Search travel options"
    },
    {
      "name": "hotel_search",
      "description": "Search travel options"
    }
  ],
  "steps": [
    {
      "type": "tool_call",
      "tool": "hotel_search",
      "arguments": {
        "city": "Tokyo"
      }
    }
  ],
  "final_answer": "Here are some hotels in Tokyo."
}

The diagnosis output is structured JSON, so the UI can render each field reliably:

{
  "failure_category": "tool_selection",
  "severity": "critical",
  "confidence": 0.96,
  "failed_step": 1,
  "root_cause": "The agent selected hotel_search for a flight request.",
  "recommended_fix": {
    "target": "tool_description",
    "before": "Search travel options",
    "after": "Search available hotels. Use only when the user explicitly requests lodging or accommodation."
  },
  "regression_test": {
    "input": "Find me a flight from Shanghai to Tokyo.",
    "expected_tool": "flight_search",
    "forbidden_tools": ["hotel_search"]
  }
}

The MVP includes: - Sample failed traces - JSON trace paste/upload - Local deterministic diagnosis engine - OpenAI-powered diagnosis route with structured output - Root cause cockpit - Evidence panel - Recommended fix diff - Regression test generator

What's Next for TraceDoctor

Next, I want to make TraceDoctor more deeply connected to the agent development loop.

The first improvement is step-linked evidence. Instead of showing evidence as plain text, each evidence item should reference the exact trace step, field, and value that supports the diagnosis. Developers should be able to click an evidence card and jump directly to the relevant tool call, tool result, or final answer.

The second improvement is executable regression evals. Today, TraceDoctor generates a regression test description. The next version should export runnable eval cases with assertions like tool_called, tool_not_called, argument_equals, and must_ask_clarification.

The third improvement is a repair-and-rerun workflow. After TraceDoctor suggests a fix, developers should be able to apply the prompt or tool description change, rerun the generated eval, and verify that the original failure no longer happens.

I also want to support richer trace formats from real agent frameworks, including model generations, tool calls, handoffs, guardrails, and custom events.

Longer term, TraceDoctor could learn from clusters of unresolved failures and propose new subcategories in the failure taxonomy. That would help teams discover recurring failure modes they did not know to look for yet.

Built With

• codex
• css
• next.js
• python
• react
• tailwind
• typescript
• vinext