Agent DriftGuard

Inspiration

Agent builders increasingly ask models to improve prompts, tools, routing, and configuration after every run. The dangerous failure is quiet: a subjective review or a tiny sample can become a permanent lesson, and the same model that proposed a change may end up validating it. Existing observability tools tell developers what happened; they rarely govern what an agent is allowed to learn from it.

What it does

Agent DriftGuard is a change-control gate for agent self-improvement. It owns the experiment instead of trusting two result files supplied after the fact:

1. Snapshot the active agent config and run a machine-verifiable baseline.
2. Lock the proposal, metric, evalset, baseline outputs, and config fingerprints before candidate results exist.
3. Apply the candidate through a config adapter and record a durable receipt.
4. Start frozen-control and candidate processes only after proposal lock, balancing which branch runs first.
5. Pair contemporaneous outcomes by task ID and require sample size, minimum lift, an exact one-sided paired-test p-value, no critical regressions, and valid evidence chains.
6. Keep the candidate or restore the last-known-good config. Candidate failures and partial config-apply failures also restore before returning.

The zero-dependency judge path launches separate control and candidate processes for every task. On the bundled 20-task evalset it moves from 8/20 to 20/20: +60 percentage points, 12 improvements, zero regressions, balanced 10/10 execution order, exact p=0.000244, and a 95% paired-bootstrap interval of +40 pp to +80 pp. DriftGuard keeps the candidate and records the full decision across four SHA-256 event chains.

The repository also includes a captured 98-call holdout replication through the Anthropic-compatible adapter using deepseek-v4-flash. A 26-task development set is separated from 36 disjoint holdout tasks. On 32 measured holdout pairs, contemporaneous control scores 0/32 and candidate scores 29/32: +90.62 pp, 29 improvements, zero regressions, exact p=1.863e-09, and a 95% paired-bootstrap interval of +78.12 pp to +100 pp. Execution order is balanced 16/16, all eight anchors pass, all response IDs are unique, and all four evidence chains verify. This is real-model execution on synthetic holdout tasks, not a production-domain or broad model-capability claim.

The representative private path uses 41 live model calls and completed human Analyst Guide v8 decisions. It moves 2/12 to 5/12 with zero regressions, but the exact paired p-value is 0.125. DriftGuard rejects the attractive result, restores the baseline config, and exports only a content-free sealed aggregate. This does not prove production lift; it demonstrates the product's central promise on real workflow evidence: unproven improvements do not become permanent config.

How we built it

The product is a Python standard-library kernel plus a zero-build local audit viewer. Its deep interface is run_experiment(proposal, evalset, runner, config, paths, policy). Runner and config adapters isolate integration-specific behavior; DriftGuard owns chronology, machine verification, pairing, policy execution, and receipts.

The command adapter exchanges JSON over standard input and output and deliberately withholds eval checks and expected answers from the agent process. The JSON config adapter writes atomically and verifies the restored baseline hash. Every experiment directory contains the raw logs, four independent hash chains, a self-contained result viewer, and the exact reproduction command.

Challenges we ran into

The hardest problem was making the evidence trail honest rather than merely visual. Append-only files alone are editable, so we added hash chains and fail-closed integrity gates. The first demo also recorded its proposal after all outcomes, which made the chronology invalid. We rebuilt the lifecycle so raw events prove baseline -> proposal lock -> config apply -> post-lock comparison -> keep or restore.

A second challenge was independence. Mixing development and confirmation tasks, or running baseline long before candidate, can create attractive but invalid lift. DriftGuard now locks disjoint evalset hashes, interleaves frozen-control and candidate execution, pairs by task ID, and rejects underpowered gains.

The private workflow added a third constraint: judges should see that a real gate ran without receiving confidential reports. The sealed export contains only hashes, counts, statistics, chain heads, and the final policy action.

Accomplishments that we're proud of

• A real candidate config is applied, then deterministically kept or restored.
• Tampered evidence cannot be folded into a release decision.
• The judge path proves the complete lifecycle without an API key or package install.
• A 98-call real-model holdout replication is checked into the repository for inspection.
• The representative private run correctly rejects a positive but statistically unproven result.
• Tests cover chronology, pairing, exact confidence gates, integrity, adapters, failure restoration, private evidence, and blind shadow operation.

What we learned

A feedback loop needs different trust levels. Soft model judgments can be useful for triage, but they should not automatically teach the system. A proposal must be locked before candidate outcomes, confirmation tasks must remain disjoint from development, and a positive delta is not enough without adequate paired evidence. The most valuable behavior is sometimes refusing to learn.

How we used Codex and GPT-5.6

Codex was the engineering environment for the Build Week work. It audited the pre-existing kernel, found the chronology flaw in the first demo, and rebuilt the lifecycle so raw events prove the complete experiment and rollback sequence. A GPT-5.6 Codex worker implemented the audit dashboard from a field-level data contract. The main GPT-5.6 Codex task then added the external-process runner, disjoint holdout flow, exact paired gate, failure restoration, private-evidence boundary, and regression tests. The demo video shows the Codex workflow and explicitly identifies the GPT-5.6 contribution.

Run it

python3 scripts/run_experiment.py \
  --proposal fixtures/orchestrated_proposal.json \
  --evalset fixtures/orchestrated_evalset.jsonl \
  --runner-command "python3 fixtures/orchestrated_command_agent.py" \
  --initialize-config fixtures/orchestrated_config.json \
  --out-dir artifacts/orchestrated-run \
  --overwrite

Open artifacts/orchestrated-run/index.html. No package installation, API key, account, network, or build step is required for this judge path. The captured real-model evidence is at artifacts/deepseek-holdout-replication-run/index.html, and the content-free representative result is at artifacts/analyst-review-sealed/evidence-summary.json.

What's next for Agent DriftGuard

Collect a fresh representative holdout with at least 20 paired outcomes, repeat the locked protocol prospectively across days, publish experiment-log heads to an external transparency store, and add signed release approvals for high-risk configurations.nspiration

Agent builders increasingly ask models to improve prompts, tools, routing, and configuration after every run. The dangerous failure is quiet: a subjective review or a tiny sample can become a permanent lesson, and the same model that proposed a change may end up validating it. Existing observability tools tell developers what happened; they rarely govern what an agent is allowed to learn from it.

What it does

Agent DriftGuard is a change-control gate for agent self-improvement. It owns the experiment instead of trusting two result files supplied after the fact:

1. Snapshot the active agent config and run a machine-verifiable baseline.
2. Lock the proposal, metric, evalset, baseline outputs, and config fingerprints before candidate results exist.
3. Apply the candidate through a config adapter and record a durable receipt.
4. Start frozen-control and candidate processes only after proposal lock, balancing which branch runs first.
5. Pair contemporaneous outcomes by task ID and require sample size, minimum lift, an exact one-sided paired-test p-value, no critical regressions, and valid evidence chains.
6. Keep the candidate or restore the last-known-good config. Candidate failures and partial config-apply failures also restore before returning.

The zero-dependency judge path launches separate control and candidate processes for every task. On the bundled 20-task evalset it moves from 8/20 to 20/20: +60 percentage points, 12 improvements, zero regressions, balanced 10/10 execution order, exact p=0.000244, and a 95% paired-bootstrap interval of +40 pp to +80 pp. DriftGuard keeps the candidate and records the full decision across four SHA-256 event chains.

The repository also includes a captured 98-call holdout replication through the Anthropic-compatible adapter using deepseek-v4-flash. A 26-task development set is separated from 36 disjoint holdout tasks. On 32 measured holdout pairs, contemporaneous control scores 0/32 and candidate scores 29/32: +90.62 pp, 29 improvements, zero regressions, exact p=1.863e-09, and a 95% paired-bootstrap interval of +78.12 pp to +100 pp. Execution order is balanced 16/16, all eight anchors pass, all response IDs are unique, and all four evidence chains verify. This is real-model execution on synthetic holdout tasks, not a production-domain or broad model-capability claim.

The representative private path uses 41 live model calls and completed human Analyst Guide v8 decisions. It moves 2/12 to 5/12 with zero regressions, but the exact paired p-value is 0.125. DriftGuard rejects the attractive result, restores the baseline config, and exports only a content-free sealed aggregate. This does not prove production lift; it demonstrates the product's central promise on real workflow evidence: unproven improvements do not become permanent config.

How we built it

The product is a Python standard-library kernel plus a zero-build local audit viewer. Its deep interface is run_experiment(proposal, evalset, runner, config, paths, policy). Runner and config adapters isolate integration-specific behavior; DriftGuard owns chronology, machine verification, pairing, policy execution, and receipts.

The command adapter exchanges JSON over standard input and output and deliberately withholds eval checks and expected answers from the agent process. The JSON config adapter writes atomically and verifies the restored baseline hash. Every experiment directory contains the raw logs, four independent hash chains, a self-contained result viewer, and the exact reproduction command.

Challenges we ran into

The hardest problem was making the evidence trail honest rather than merely visual. Append-only files alone are editable, so we added hash chains and fail-closed integrity gates. The first demo also recorded its proposal after all outcomes, which made the chronology invalid. We rebuilt the lifecycle so raw events prove baseline -> proposal lock -> config apply -> post-lock comparison -> keep or restore.

A second challenge was independence. Mixing development and confirmation tasks, or running baseline long before candidate, can create attractive but invalid lift. DriftGuard now locks disjoint evalset hashes, interleaves frozen-control and candidate execution, pairs by task ID, and rejects underpowered gains.

The private workflow added a third constraint: judges should see that a real gate ran without receiving confidential reports. The sealed export contains only hashes, counts, statistics, chain heads, and the final policy action.

Accomplishments that we're proud of

• A real candidate config is applied, then deterministically kept or restored.
• Tampered evidence cannot be folded into a release decision.
• The judge path proves the complete lifecycle without an API key or package install.
• A 98-call real-model holdout replication is checked into the repository for inspection.
• The representative private run correctly rejects a positive but statistically unproven result.
• Tests cover chronology, pairing, exact confidence gates, integrity, adapters, failure restoration, private evidence, and blind shadow operation.

What we learned

A feedback loop needs different trust levels. Soft model judgments can be useful for triage, but they should not automatically teach the system. A proposal must be locked before candidate outcomes, confirmation tasks must remain disjoint from development, and a positive delta is not enough without adequate paired evidence. The most valuable behavior is sometimes refusing to learn.

How we used Codex and GPT-5.6

Codex was the engineering environment for the Build Week work. It audited the pre-existing kernel, found the chronology flaw in the first demo, and rebuilt the lifecycle so raw events prove the complete experiment and rollback sequence. A GPT-5.6 Codex worker implemented the audit dashboard from a field-level data contract. The main GPT-5.6 Codex task then added the external-process runner, disjoint holdout flow, exact paired gate, failure restoration, private-evidence boundary, and regression tests. The demo video shows the Codex workflow and explicitly identifies the GPT-5.6 contribution.

Run it

python3 scripts/run_experiment.py \
  --proposal fixtures/orchestrated_proposal.json \
  --evalset fixtures/orchestrated_evalset.jsonl \
  --runner-command "python3 fixtures/orchestrated_command_agent.py" \
  --initialize-config fixtures/orchestrated_config.json \
  --out-dir artifacts/orchestrated-run \
  --overwrite

Open artifacts/orchestrated-run/index.html. No package installation, API key, account, network, or build step is required for this judge path. The captured real-model evidence is at artifacts/deepseek-holdout-replication-run/index.html, and the content-free representative result is at artifacts/analyst-review-sealed/evidence-summary.json.

What's next for Agent DriftGuard

Collect a fresh representative holdout with at least 20 paired outcomes, repeat the locked protocol prospectively across days, publish experiment-log heads to an external transparency store, and add signed release approvals for high-risk configurations.

Built With

• codex
• gpt-5.6
• javascript
• openai
• python