Agentic Coding Evaluation (ACE)

ACE is a reproducible benchmark for coding agents: not just pass/fail, but effort-adjusted scoring of how hard a task fought to succeed. Runs Nebius Token Factory + NVIDIA Nemotron as agent and judge.

Comment

Inspiration

A model says "tests pass," and plenty of coding-agent benchmarks just believe it — even when the tests were gutted or the build never actually ran. We didn't want another leaderboard. We wanted an answer to a narrower question: for a real task, which model, harness, and settings actually get the job done most cost-effectively — graded by something built to catch us if we tried to cheat it ourselves, down to mutating the arithmetic nearest the task's own money calculation and checking whether the agent's own tests would even notice.

What it does

Built by StrGm AI — a startup founded to turn ideas like this into real products — Agentic Coding Evaluation (ACE) drives a coding agent through a frozen build task — a small multi-service trading platform (Spring Boot, PostgreSQL + Flyway, React, Docker Compose) — and scores it with an independent oracle that never takes the agent's word for anything. Thirty checks, 77 points, fixed denominator: an attempt the agent skipped is a FAIL, not a silent gap.

The oracle backs that up with real evidence, not surface signals. It parses JUnit XML instead of trusting an exit code. It rebuilds the compose stack from a source-only copy so a prebuilt jar can't sneak into an image. It mutates the arithmetic nearest the task's own money calculation and checks that the agent's own test suite would actually catch the break. Six adversarial "cheat" fixtures keep the oracle itself honest. A task ladder (L1 … L7_full_platform) scales the ask, and a provider registry routes any job to local hardware or an external endpoint — which is where Nebius Token Factory and NVIDIA's Nemotron come in.

How we built it

ace-service (Java 21 / Spring Boot) runs the job queue, benchmark runner, and oracle; ace-ui-vaadin is its front end. For the hackathon, job ran its coding-agent session against nebius/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B over Nebius Token Factory — the model under test, not a demo stand-in. The same model also checkpoints itself mid-task as the judge, reading the task's own goal and trajectory every few turns. Separately, a second model on the same Nebius endpoint (zai-org/GLM-5.3-Flash) sits in the independent reviewer role — grading the agent's full trajectory after the fact, with its verdict recorded under its own identity rather than folded into the correctness score, the one voice in the loop that isn't marking its own homework.

A session that runs out of budget or blows its context window can get one rescue attempt — opt-in per job, not automatic — a trimmed-history reconnect on its own separate budget, before the harness accepts that final state.

Challenges we ran into

Getting Docker to behave was harder than expected. We first tried making it fully automatic — starting it when the agent needed it, shutting it down when idle — but that shutdown sometimes fired in the middle of a build, silently killing work in progress. We simplified: the operator starts Docker, and ACE just uses whatever's already running.

Pinning the toolchain mattered more than it sounds. The agent and the grader need the exact same Java version, not two different ones — so before any job starts, we run a real build against a known-good project to confirm the environment actually works. If that check fails, it's recorded as an environment problem, never as the agent's own failure.

Telling a real rate limit apart from our own decision to cut the agent off was a subtler one. Both look identical from the outside, and treating every one as "the agent ran out of budget" would have unfairly blamed the model for what was really a provider hiccup — so we made sure the two cases are distinguished by what actually caused them, not just how they look.

Accomplishments that we're proud of

Our grader holds up under real cheating attempts, not just a clean demo run: every deliberate attempt we built to fool it — hidden shortcuts, swapped-in prebuilt code, sneaky test tricks — still gets caught, while a genuinely good solution still passes comfortably. That's the real test of a grader: not just that it's strict, but that it's strict about the right things.

Our safety net for a struggling task did real work this hackathon, not just in a test case: when a task ran out of room partway through, it got one more dedicated attempt and came back with a clear, useful account of what it had built and what was still missing, instead of nothing at all. And when a production hiccup actually knocked out the live benchmark worker mid-task, simply restarting the service was enough to fix it cleanly — the job picked itself back up with no manual cleanup needed. (A person still had to notice something was wrong and restart it — this isn't a system watching itself, just one that never leaves a job stranded once someone steps in.) The same "never trust, always verify" discipline we built into the grader turned out to make the whole system easier to recover, too.

What we learned

Building a grader that refuses to trust an agent's self-report taught us to apply the same discipline to our own infrastructure. The most dangerous failures weren't loud — a disk silently filling mid-run didn't crash the job, it wedged the one worker thread that runs every queued job, with zero exception logged; recovery only came from noticing the silence itself, not an error. The agent's own "done" is exactly as unreliable as its own "blocked": one run reported a blocked build when the real cause was a one-line path mistake in a wrapper script it had written itself minutes earlier. If we hadn't gone looking, we'd have scored the symptom instead of the cause. That's the whole thesis of ACE in miniature — "looks finished" and "is finished" are different claims, and almost all the real engineering effort goes into closing that gap, not widening it.

What's next for Agentic Coding Evaluation (ACE)

Right now, finding the winning configuration still takes a human: set up an Experiments batch, wait for every arm to finish, then read Compare or the Groups leaderboard by hand. The next step is to close that loop — an agent that treats ACE's own scoring as its reward signal and tunes the coding agent's own knobs (model choice, harness mode, compaction strategy, context window, budget split) autonomously, run after run, converging on whichever configuration gets a given task done best for the cost it actually took. Effort-adjusted scoring and per-call dollar-cost accounting are the prerequisites already in motion; agentic parameter tuning is what turns ACE from a benchmark you read into an optimizer that reads itself — and the first thing worth tuning against is which Nebius-hosted model gets the most done per dollar.

Built With

Share this project:

Updates

Submission history