EvalForge

• 

  EvalForge turns coding-agent quality into reproducible weak-versus-fixed evidence.

Inspiration

Coding agents and reusable skills are becoming easy to create, but evaluating them still starts with a blank page. Developers must invent realistic tasks, build fixtures, write graders, reset workspaces, and collect enough evidence to explain failures. As a result, many agent demos rely on happy paths or an opaque LLM quality score.

I wanted the evaluation itself to become a reusable engineering artifact.

What it does

EvalForge reads a SKILL.md, issue specification, and fixture inventory. Codex with GPT-5.6 generates a three-case evaluation suite covering requirement completion, regression coverage, and scope control. Structured Outputs constrain the suite, then a local policy layer rejects unsafe commands, absolute paths, traversal, unknown fields, and unprotected evaluator files.

The suite runs in a clean Git workspace. Deterministic graders inspect test exit codes and changed files. EvalForge exports JSON and Markdown reports that show why each case passed or failed.

The demo applies the same suite to a weak and fixed issue-fixing skill. The weak version scores 0/3 because it misses a username boundary, adds no regression test, and modifies an unrelated README. A real Codex GPT-5.6 Sol coding run passes all three checks.

How I built it

The CLI and evaluation engine are written in Python with no runtime dependencies. The evidence console is a responsive React/vinext site. Codex CLI provides a read-only suite-generation adapter and a workspace-write coding-agent adapter. Generated suites are constrained by JSON Schema and validated again before execution.

Codex and GPT-5.6 were also my development environment throughout Build Week. They helped narrow the product scope, implement the vertical slice, discover a nested-sandbox failure, add the infrastructure-error boundary, generate and compare suites, build the UI, and verify the final package.

Challenges

The hardest problem was preventing the evaluator from becoming circular: an LLM cannot simply invent a task and then award its own score. EvalForge therefore uses GPT-5.6 for test design but ordinary code for the final evidence wherever possible.

Another challenge was distinguishing an agent failure from a runner failure. When a nested macOS sandbox blocked every tool call, EvalForge learned to emit infrastructure_error instead of turning an invalid run into a quality judgment.

A real GPT-5.6 Sol Codex run also exposed an evaluator false negative: Codex added test_app.py, while the generated contract initially accepted only files under tests/. Inspectable evidence made the mistake obvious. I broadened the contract, added regression coverage for both layouts, and regraded the exact same frozen workspace at 3/3.

Accomplishments

• Codex-generated, human-editable eval suite
• Strict command and path safety policy
• Clean, repeatable fixture workspaces
• Evidence from tests and Git changes rather than one subjective score
• Weak 0/3 versus real Codex 3/3 comparison using one portable suite
• Three Codex generations with stable executable grader behavior
• Responsive evidence console with no live-model latency
• Fourteen automated tests and a production deployment

What I learned

Agent evaluation needs two independent layers: creative scenario design and deterministic verification. Evaluator infrastructure must also be observable, or sandbox failures, timeouts, and missing credentials can be mistaken for agent defects.

What's next

Next I would add generated Docker fixtures, more language-specific grader templates, editable approval of generated cases, GitHub Actions export, and a catalog of real-world coding-skill benchmarks. EvalForge could also produce evaluation objectives for optimization tools without becoming an optimizer itself.

Built With

• git
• gpt-5.6
• json-schema
• openai-codex
• python
• react
• typescript