HarnessLab

• 

  what broke

Inspiration

Software teams regression-test the code they ship, but increasingly that code is being written by AI coding agents.

Those agents have their own configuration: model, system prompt, reusable skills, context policy, tools, permissions, and execution limits. When several of those change and an agent suddenly performs worse, it can be extremely difficult to tell what actually caused the regression.

HarnessLab asks a simple question:

Can we regression-test the coding-agent harness itself?

What it does

HarnessLab compares a trusted baseline coding-agent configuration against a candidate configuration and runs controlled coding experiments to investigate what changed.

It:

• detects which harness dimensions changed
• runs experiments from controlled repository states
• evaluates results using deterministic software evidence
• adaptively chooses configurations that help eliminate competing explanations
• repeats important experiments to detect stochastic behavior
• reports a diagnosis only when the evidence supports one
• returns INCONCLUSIVE when it cannot confidently isolate the cause

The goal is not to make an LLM guess why another LLM failed.

The principle is:

LLM reasons; deterministic software proves.

How we built it

The coding agent uses NVIDIA Nemotron 3 Super 120B A12B through Nebius Token Factory.

Experiments run in isolated Nebius Token Factory Sandboxes. HarnessLab prepares a repository, creates a shared checkpoint, and forks experiments from the same starting state so the task and code remain controlled while the harness configuration changes.

HarnessLab records evidence including:

• build and test results
• hidden-test results
• files changed
• commands and tool calls
• termination behavior
• repeated-run outcomes

The diagnosis layer then uses controlled experiments to reduce the possible explanations for a regression.

The application is built with Next.js, TypeScript, Python, Node.js, and the Contree SDK.

A regression we discovered

During development, an earlier controlled experiment exposed an interaction between an agent Skill and a Compact Context policy.

Each change worked independently, but together they caused the agent to repeatedly request information that was no longer available in its compacted context. An optimization that avoided re-sending unchanged files then returned short markers rather than the original content.

The agent kept retrying until it exhausted its turn budget without implementing the task.

That result demonstrated why interactions between individually reasonable harness changes can matter.

This example comes from our earlier single-run evidence schema and is clearly labeled as such in the demo.

The harder problem: stochasticity

Real experiments exposed another challenge: the same coding-agent configuration does not always produce the same result.

A configuration may pass one run and fail another.

That changed the direction of HarnessLab.

The current version repeats important experiments, records stability and failure rates, and can refuse to make a root-cause claim when the evidence is not strong enough.

Our current benchmark includes cases where HarnessLab deliberately reports an inconclusive result rather than producing a false-confident diagnosis.

We consider that behavior a feature, not a failure.

Challenges

The hardest parts were:

• isolating harness changes from repository changes
• running real coding agents inside remote sandboxes
• building reproducible experiment forks from shared checkpoints
• separating deterministic evaluation from model-generated explanations
• dealing honestly with stochastic agent behavior
• designing an experiment planner with a limited run budget

What we learned

The biggest lesson was that debugging coding agents is not just an LLM reasoning problem.

It is an experimentation problem.

Once agent behavior becomes stochastic, a single PASS or FAIL is not enough evidence to confidently assign causality.

Infrastructure for coding agents will need the same ideas software engineering already uses elsewhere: controlled experiments, regression testing, replication, reproducibility, and uncertainty.

What's next

HarnessLab could evolve into a CI system for coding-agent configurations.

A future version could connect directly to repositories and automatically test changes to prompts, skills, context strategies, models, tools, and execution policies before those changes reach production.

Teams already regression-test their software.

HarnessLab brings that same discipline to the agents writing it.we're proud of

Built With

• contree-sdk
• nebius-token-factory
• nebius-token-factory-sandboxes
• nemotron-3-super
• next.js
• node.js
• nvidia-nemotron
• python
• tailwind-css
• typescript
• vercel