The Asymmetric Ratchet: What a 4-Arm Factorial Revealed About Multi-Agent Refinement
In my previous update, I argued that we needed to isolate structured evidence from progressive escalation before attributing improvements to either one.
I’ve now completed Replicate 1 of a (2 \times 2) factorial experiment across the same 50-seed C security benchmark (25 benchmark-labelled safe, 25 vulnerable), using Gemini 3.5 Flash for debate and Gemini 3.6 Flash for judging and verification on Gemini Enterprise Agent Platform.
The four arms were:
- A — Raw, no escalation: raw code, single-round debate.
- B — Structured, no escalation: structured evidence, single-round debate.
- C — Structured + escalation: structured evidence with progressive refinement.
- D — Raw + escalation: raw code with progressive refinement.
I then joined the terminal decisions against our Track 1 independent-adjudication ledger. Of the 50 seeds, 24 currently have resolved security verdicts: 13 proved safe and 11 confirmed vulnerable. These are preliminary, single-replicate findings—not a final estimate of general performance.
1. Escalation improved recall, but increased inference cost
| Arm | Condition | Recall on confirmed-vulnerable seeds | False-positive rate on proved-safe seeds | F1 | Accuracy on resolved seeds |
|---|---|---|---|---|---|
| A | Raw baseline | 63.6% (7/11) | 23.1% (3/13) | 0.667 | 0.708 |
| B | Structured evidence | 72.7% (8/11) | 23.1% (3/13) | 0.727 | 0.750 |
| C | Structured + escalation | 81.8% (9/11) | 23.1% (3/13) | 0.783 | 0.792 |
| D | Raw + escalation | 81.8% (9/11) | 23.1% (3/13) | 0.783 | 0.783* |
*Arm D has 49 recorded outcomes because Seed 50 timed out. Its reported accuracy uses the available outcomes; it should not be compared directly with the 50-seed arms without accounting for the missing result.
On this adjudicated subset, structured evidence alone was associated with a one-case increase in accepted true vulnerabilities. Adding escalation to structured evidence increased recall from 63.6% to 81.8%, while the number of accepted claims on proved-safe seeds remained three out of thirteen.
Across the full benchmark-labelled cohort, Arm C accepted 25/25 vulnerable-labelled seeds. That is 100% acceptance on benchmark labels, not proof that all 25 claims were correct. Independent adjudication remains incomplete for the full manifest.
Structured evidence also showed a preliminary filtering signal: acceptance on four triage-labelled likely-invalid seeds fell from 75% in Arm A to 50% in Arms B, C and D. The cohort is too small to support a stable general claim.
Escalation was expensive. Arm C averaged approximately 66,305 tokens per seed and cost $11.86 across the run, compared with 30,356 tokens per seed and $5.49 for Arm B. In this replicate, the higher recall came with substantially greater inference consumption.
2. The Con path was winning debates—but not producing accepted defenses
The terminal ledger initially showed zero accepted Con defenses. That did not mean Con never won a debate.
I audited all 421 attempt records across the four arms. The logs recorded 12–17 Con wins per arm, but every recorded Con win had an invalid formal receipt contract. Consequently, these outcomes were rejected as con_win_not_applicable, rather than accepted as deterministically established defenses.
The distinction matters:
- Debate outcome: which side the judge favoured.
- Receipt validity: whether that result met the machine-verifiable contract.
- Terminal decision: whether the runtime ultimately accepted the claim or defense.
The verifier also intercepted Pro-side logic errors during the run. Arm C recorded 111 attempt-level interceptions, while the final accepted corpus had zero observed INV-1 logic errors.
That is evidence that the gate rejected problematic attempts under the tested invariant. It does not establish that every accepted security claim was correct, or that the system had zero security errors of every kind.
3. The discovery: a potential asymmetric escalation ratchet
The attempt-level audit exposed a more consequential question about the runtime's control flow.
The observed sequence appears to be:
- Pro wins Round 0: the verifier passes the claim, and the episode stops successfully.
- Con wins Round 0: the Con receipt fails the formal contract, so the result is treated as an unsuccessful episode.
- Escalation triggers: the reflector initiates another round, giving Pro another opportunity to win.
- Pro wins Round 1: if the claim passes the verifier, it can become the terminal accepted result.
This creates a potential asymmetry. A Pro win can end the episode, while a Con win without a valid receipt can trigger another opportunity for Pro.
The issue is not the terminal-attempt analysis rule. Recording the actual final attempt is consistent with the experiment specification. The concern is the runtime policy that determines which outcomes stop execution and which outcomes trigger another round.
Seed 47: a concrete diagnostic case
Seed 47 concerns an alleged use-after-free involving GFS2 ACL handling.
Our Track 1 audit found that posix_acl_from_xattr allocates its own ACL structure, refuting the specific vulnerability predicate under review.
In Arm C, the attempt history reportedly shows:
- Con argued that the alleged use-after-free was not supported by the allocation behaviour.
- The judge favoured Con in Round 0.
- The Con receipt failed the deterministic contract.
- The runtime treated the result as a failed episode and escalated.
- Pro won Round 1, and the claim was accepted.
If confirmed against the raw attempt records and runtime trace, this is a concrete example of a counterargument that aligned with the independent audit but could not influence the terminal result because its receipt was not machine-verifiable.
Related sequences were reported for Seeds 51, 77, 87 and 88 in Arm C. These cases need to be examined individually; they should not be assumed to have identical security significance.
4. What this teaches us about agentic safety
Iterative refinement is not automatically neutral. A second reasoning round can recover a missed vulnerability—but it can also give a previously challenged claim another chance to win.
If one side's success terminates an episode while the other side's uncertifiable success triggers another attempt, the runtime can favour one outcome even when the debate itself produces useful counterevidence.
This is not yet proof that asymmetric escalation explains all of the observed acceptance increase. It is a mechanism identified in one replicate that can be tested directly.
For BARRED, the engineering implications are clear:
- Con needs a machine-verifiable receipt format that can express scoped counterevidence.
- A receipt-contract failure must not be confused with proof that the counterargument is wrong.
- Escalation should explicitly address the unresolved dispute rather than automatically give Pro another opportunity.
- The runtime should log debate winners, receipt validity, verifier results, retry triggers and terminal decisions separately.
- Correctness must be measured against independently adjudicated security outcomes, not acceptance yield alone.
The next step is to test the control-flow mechanism and improve the receipt contract before interpreting further escalation results as evidence of better security reasoning.
The broader research question remains: how much of a model's reasoning burden can deterministic evidence discharge—and how do we ensure the runtime preserves that evidence instead of discarding it at a decision boundary?
This remains ongoing post-submission research. The original hackathon submission remains the official competition snapshot.
Log in or sign up for Devpost to join the conversation.