Inspiration
AI coding agents are improving quickly, but evaluating them on toy problems does not te us whether they can handle real software engineering work. We created TraceBench to tur real pull requests-with authentic requirements, repository history, and tests-into reproducible, secure benchmark tasks.
What it does
TraceBench converts real-world pull requests into isolated Harbor tasks for evaluating coding agents. Each task includes: • The repository at its pre-PR state • The original pull-request description • Relevant historical agent traces • A verified reference solution • Behavioral tests and granular scoring • A network-restricted sandbox that prevents agents from downloading the answer TraceBench can run the same tasks across different models and agent harnesses, then produce comparable scores, trajectories, and aggregate results.
How we built it
We built a reproducible pipeline that:
- Loads pull-request metadata and execution traces from the TraceBench corpus.
- Identifies the repository state immediately before each pull request.
- Reconstructs a secure Git repository with real history but no remotes.
- Generates the task instructions, environment, verifier, and reference solution.
- Validates the oracle solution before accepting the task.
- Runs multiple attempts through Harbor using containerized sandboxes.
- Records per-task rewards, agent trajectories, token usage, and aggregate results. The loader and orchestration layer are written in Python, while many benchmark targets and supporting tools use Go. Docker provides repeatable execution environments, Harbor
Log in or sign up for Devpost to join the conversation.