Inspiration

AI agents can look impressive in demos while failing silently in real workflows. Teams often have traces, logs, and isolated test cases, but no repeatable way to turn them into evidence, regression tests, and prioritized fixes.

What it does

EvalForge is a reproducible evaluation and failure-diagnosis workbench for AI agents. It is designed to:

  • ingest multi-step agent traces and tool calls;
  • convert production failures into replayable evaluation cases;
  • score correctness, reliability, latency, and cost;
  • cluster failures by root cause instead of surface error text;
  • generate evidence-backed reports and regression tests for developers.

How we are building it

The MVP uses a modular pipeline: trace ingestion, normalized event storage, deterministic replay, evaluator plugins, failure classification, and an interactive report. The architecture keeps model judging separate from hard assertions so every score can be traced back to evidence.

Challenges

The hardest problems are preserving trace boundaries across providers, distinguishing agent mistakes from tool or infrastructure failures, and making results reproducible when model outputs are stochastic.

What we learned

Agent quality is not one score. Useful evaluation requires explicit contracts, provenance, repeatable inputs, and failure categories that map to engineering changes.

What's next

During the hackathon, we will deliver an end-to-end MVP with trace upload, configurable evaluators, regression comparison, failure clustering, and a shareable report.

Built With

Share this project:

Updates

Submission history