Inspiration
Public coding benchmarks like SWE-bench tell you how an agent does on someone else's repos. They don't tell you whether an agent will actually work on your codebase. I build benchmark tasks for AI coding agents, and the hardest part is always the same: hand-authoring tasks that are realistic, verifiable, and not trivially solvable. That doesn't scale. Gauntlet automates it.
What it does
Gauntlet takes any repository and turns it into a custom benchmark for coding agents:
- Generate: it analyzes the repo and produces realistic software engineering tasks (bug fixes, features, refactors) with instructions and tests.
- Validate: each task is checked before it's trusted. The tests must fail on the unmodified repo and pass on a reference solution.
- Evaluate: coding agents attempt the validated tasks in a sandbox, and Gauntlet scores each attempt on verifiable outcomes.
- Export: tasks are exported in Harbor format, so they plug into existing agent-evaluation pipelines.
The result is a ranking of agents and models on your code, not a generic leaderboard.
How I built it
- Models: NVIDIA open-source models served through Nebius Token Factory for task generation, solution drafting, and agent attempts.
- Orchestration: a Python-first pipeline with separate stages for generation, validation, attempts, and scoring.
- Grounding with Tavily: an ablation compares task quality with and without web-search context, to measure whether external knowledge makes tasks more realistic or just leakier. Result:
[your finding]. - Observability with LangSmith: every stage (task generation, validation, agent attempts, tool calls, scoring) is traced. When a run fails, I can see exactly where, before spending more credits.
- Harbor export:tasks are packaged in the format used by agent-evaluation platforms.
Challenges
- Trivially solvable or unsolvable tasks. An LLM-written task is often either leaked in its own description or broken. Execution-based validation (fail-before, pass-after) filtered these out:
[X of Y]generated tasks survived. - Flaky tests. Tests that pass sometimes made scoring meaningless, so validation reruns each task multiple times.
- Debugging multi-stage agent runs. Failures could come from generation, the sandbox, the agent, or the scorer. LangSmith traces made this tractable.
- Credit budget. Tracing and caching let me iterate without burning through tokens.
What I learned
Task validation matters more than task generation. A generator is easy to build, but a benchmark is only as trustworthy as its filtering. I also saw how differently agents behave across repos, which is exactly why repo-specific evals are useful.
What's next
- Support for more languages beyond Python
- Difficulty calibration from agent pass rates
- CI integration, so every new agent or model version is evaluated on your repo automatically
Log in or sign up for Devpost to join the conversation.