AgentWorkbench - Arena

Developers today have more choices than ever when using AI coding tools. The question is no longer simply "Which model is best?" My actual coding workflow depends on an entire configuration: the harness, provider, reasoning level, permissions, tools, and execution environment.

Public leaderboards are valuable, but they cannot tell me which complete configuration will perform best on my own repository and my own tasks. And manually going through by hand to try and work all this out is a long and tedious process.

Arena was created to answer that question. Rather than benchmarking models in isolation, Arena compares complete coding-agent configurations on the same repository, starting from the same Git baseline and working under the same task contract. My goal is to produce practical evidence that helps developers choose the right configuration for the job.

Arena is also the first subsystem planned for a larger side project of mine, where specialized development agents will eventually be assigned configurations based on accumulated evidence instead of assumptions.

What it does

Arena executes the same software task across multiple native coding-agent configurations.

Each candidate receives:

The same repository baseline The same task contract The same validation rules The same execution limits Its own isolated Git worktree

Arena then:

Applies deterministic hard gates such as tests, file boundaries, dependency policies, timeouts, and evidence completeness. Records operational telemetry including execution time, reported token usage, retries, tool calls, interventions, and code-change size. Presents only the eligible candidates to GPT-5.6 Sol using identity-masked evaluation packets. Produces a recommendation explaining both semantic quality and operational tradeoffs.

The result is a repository-specific comparison of complete AI coding configurations rather than a generic benchmark.

How I built it

The project was designed first as a product rather than simply as a benchmark.

I defined the comparison model, deterministic validation policy, telemetry schema, identity-masking process, evaluation flow, and long-term integration into AgentWorkbench.

Using Codex as the primary implementation partner, I built the project through iterative development of:

Native Codex and OpenCode adapters Isolated Git worktree execution Trial definitions Deterministic validation Evidence collection Telemetry normalization GPT-5.6 adjudication Static reporting

Throughout development I intentionally separated objective evidence from subjective evaluation. GPT-5.6 can recommend among eligible candidates, but it cannot override deterministic failures such as failed tests or policy violations.

Challenges I ran into

One was definitely defining what "fair" actually means when comparing AI coding agents.

Different harnesses expose different telemetry, providers report usage differently, and some information simply is not available everywhere. Arena avoids inventing missing values by explicitly distinguishing "Not reported" from zero.

Another challenge was avoiding misleading conclusions. Arena compares complete configurations, not isolated models. That distinction became a core design principle because changing multiple variables at once makes it impossible to honestly claim that a single model caused the observed result.

Finally, I wanted GPT-5.6 to evaluate implementation quality without introducing brand bias. That led to the identity-masked adjudication system, where semantic evaluation occurs before candidate identities are revealed.

What I learned

This project reinforced that evaluation systems need clear boundaries between objective facts and subjective judgment.

Deterministic validation provides trustworthy evidence about whether requirements were satisfied. Large language models excel at evaluating qualities like maintainability, architectural fit, and unnecessary complexity. Combining those strengths produces more useful recommendations than relying on either approach alone, though admittedly I think I may have been more than a bit haphazard about implementing this properly.

I also learned that building reliable developer tooling is often more about defining good evaluation policies than writing large amounts of code.

What's next for AgentWorkbench Arena

Arena is intended to become the calibration engine for AgentWorkbench.

As more repository-specific trials are collected, AgentWorkbench will use Arena's accumulated evidence to recommend the most appropriate configuration for specialized development roles such as:

Explorer Coder Reviewer Debugger Researcher

Instead of assigning configurations based on reputation or public benchmarks, future AgentWorkbench agents will earn those assignments through demonstrated performance on my preferred development work.

Built With

• 5.6
• agents
• automation
• cli
• codex
• git
• github
• gpt
• html
• node.js
• opencode
• telemetry
• typescript
• yaml