Timeless Harness

An evidence-led governor that tests reusable Agent workflows against a native baseline and returns a provisional keep, narrow, revise, retire, or unresolved decision.

Comment

Inspiration

Modern coding Agents are already capable. They plan, use tools, test, review, and repair their work. A reusable workflow can help, but it can also slow the model down, create rituals, and introduce new mistakes.

Timeless Harness began as a broad method for improving every Agent task. Its own evaluation contradicted that idea. The older wrapper won fewer recorded pair preferences than the native condition and added unnecessary process. I chose to remove the failed design instead of protecting it.

That left one useful question:

Does this reusable Agent workflow still earn its cost?

What it does

Timeless Harness governs Skills, instructions, memory rules, evaluators, and repair methods. It does not wrap ordinary work.

It has two parts:

  1. A small Skill gives an Agent a lifecycle policy: keep, narrow, revise, retire, or unresolved.
  2. A Python Governor makes native-versus-workflow comparisons inspectable.

The Governor can:

  • freeze the task, Skill, rubric, starting workspace, evaluator, and cost rule before outputs exist;
  • run a native baseline and workflow treatment with the same model settings;
  • keep the two executions in separate random temporary roots;
  • hash captures, telemetry, final workspaces, file contents, and file modes, then detect later changes;
  • reject missing telemetry, non-finite cost rules, sensitive workspace files, and unsupported sandbox access;
  • prepare a condition-label-blind packet containing answers, artifact manifests, and optional evaluator results;
  • combine reviewer quality with tokens and wall time;
  • return a clearly labeled provisional lifecycle candidate.

A single pair is never presented as proof. Final decisions require repeated representative cases and a stopping rule chosen before seeing results.

The self-audit

The public repository includes the complete evidence that caused Timeless to retire its first design:

  • 4 fixed tasks;
  • 12 paired trials;
  • 24 outputs with opaque IDs;
  • 72 rubric score rows;
  • 36 recorded pair preferences from three grader result sets;
  • grader notes, revealed mapping, and SHA-256 checksums.

Anyone can run one zero-credit command to verify 44 files and recompute the result:

Native baseline: 19 wins, 3.991 mean score, 0 unnecessary-process flags
Old Timeless wrapper: 6 wins, 3.889 mean score, 9 unnecessary-process flags
Fatal omissions: 0 in both conditions
Ties: 11
Decision: retire tested general wrapper

The recurring failure was concrete. The old wrapper added an arbitrary three-pass release ritual without improving the answer. That result removed routine invocation, fixed process stages, persistent workspace ceremony, and general superiority claims.

The evidence supports retiring the tested old wrapper. It does not prove that the new narrow Governor improves unseen tasks. That claim remains unresolved and is stated plainly in the product.

How I built it

The Governor uses only the Python standard library. The judge path makes no model call and needs no credentials.

Codex was the main engineering environment. I used it to reverse the submission requirements into acceptance checks, implement the Governor, test failures, inspect the recovered raw evaluation package, audit privacy, simplify the repository, and keep the README, Skill, CLI, evidence, and submission story consistent.

GPT-5.6 Sol with ultra reasoning was used through Codex for implementation, adversarial review, evidence analysis, security review, and product critique. Separate fresh-context review passes found real defects, including a NaN cost-policy bypass, mutable run evidence, incomplete workspace hashing, weak condition separation, and a misleading synthetic Demo. Those defects were repaired, covered by tests, or removed.

The human author set the product goal, privacy boundary, evaluation policy, and final release decision.

Challenges

The first challenge was resisting the urge to make a philosophy look like a product. The useful part had to change a real decision.

The second was using a strong baseline. Timeless should not win because the native Agent was artificially restricted. The prospective runner therefore fixes the model settings and starting artifacts, while the treatment changes only the reusable Skill.

The third was keeping the evaluation honest. Hashes are not signatures. Separate temporary roots are not operating-system isolation. A blind grader can still be wrong. The repository states these limits instead of turning risk reduction into a security claim.

The fourth was allowing deletion to count as progress. The best feature of the first prototype was the evidence that showed it should not survive.

What I learned

A workflow should not survive because it sounds wise, took effort to create, or makes an Agent look busy.

As models improve, instruction debt becomes a product problem. Every reusable rule should have a claim, evidence, scope, cost, and retirement condition.

Timeless Harness does not promise to make every Agent smarter. It makes the outer workflow earn the right to stay.

A tie with more process is a loss.

Built With

Share this project:

Updates