Agentic Harness

Independent verification for Codex and other coding agents—before work is marked done.

Comment

Inspiration

Coding agents can produce impressive work, but they can also say “done” before the requested outcome actually works. I built Agentic Harness around one simple idea:

A coding agent saying “done” is not proof that the task is done.

I wanted a practical supervisor that lets developers use agents such as Codex while making completion depend on independent evidence—not confidence, prose, or a successful-looking status message.

What it does

Agentic Harness runs a coding agent against one project-local goal, records its work, and checks the result using an independent verification command chosen by the user.

If verification fails, the harness can return the failure to the agent for another attempt. If the agent cannot complete the task within its limits, the result is marked blocked or failed—not silently converted into success.

Users can work through either:

  • A browser interface with setup, live progress, changed files, checks, retries, and final evidence
  • A command-line interface for automated and developer workflows
  • Packaged recipes for tests, linting, type checking, documentation, and changelog work

Each run produces a durable project-local report showing what changed, which checks ran, and why the result was accepted or rejected.

How I built it

Agentic Harness is distributed as one Python package containing a shared execution engine, CLI, browser interface, project-state model, retry loop, evidence contract, redaction controls, and independent completion gate.

For Build Week, I used Codex with GPT-5.6 to help develop and review the product workflow, strengthen the verification boundary, improve the first-run browser experience, expand cross-platform testing, and challenge assumptions about what should count as trustworthy evidence.

Codex was especially useful for working across the engine, browser UI, tests, packaging, documentation, and release automation while keeping the central product rule consistent: worker-authored completion claims are untrusted until an independent check passes.

Challenges

The hardest challenge was preventing the verification system from becoming another source of false confidence. A passing process, generated report, or agent-written explanation cannot automatically prove the requested behavior.

Other challenges included:

  • Preserving evidence across retries and interrupted runs
  • Detecting workspace changes without accepting stale results
  • Keeping secrets and private paths out of reports
  • Supporting different coding agents and OpenAI-compatible models
  • Making a powerful supervision system understandable to first-time users
  • Testing honestly without presenting controlled experiments as broad model-quality claims

What I learned

I learned that supervising agents is less about adding more prompts and more about defining a trustworthy completion contract.

Verification must be independent, current, reproducible, and tied to the original objective. Resource limits are also not success conditions: when time, token, or retry budgets are exhausted, the honest result is blocked—not done.

I also learned that product clarity matters as much as technical depth. The most useful experience is simple: select a project, describe the outcome, provide the command that proves it, and let the harness show whether the work is genuinely verified.

Accomplishments

Agentic Harness v0.13.1 is publicly available through GitHub and PyPI and includes both the CLI and browser interface. The current release uses cross-platform CI along with automated type, lint, packaging, and browser checks.

A controlled 24-case evaluation demonstrated the completion gate catching premature claims and retrying repairable tasks. A harder preregistered Codex comparison also produced an important honest result: both approaches passed 9 of 10 verifiers, but direct execution falsely accepted the missed task while Agentic Harness refused to mark it complete.

What’s next

Next I want to test Agentic Harness on more real, messy repositories and make the first successful run even simpler. The long-term goal is straightforward:

Let developers use powerful coding agents while making “done” mean independently verified.

Built With

Share this project:

Updates