Gemini 3 changed the question
After the progressive-escalation experiments, I ran a small comparative pilot to see what happens when the BARRED debate swarm moves from the local Gemma stack to Gemini 3 on Vertex AI.
This was only a 10-seed pilot — 5 Safe and 5 Vulnerable — so I am not treating it as a model ranking.
What it did reveal was more interesting.
The acceptance asymmetry
Across the pilot:
| Stack | Vulnerable accepted | Safe accepted |
|---|---|---|
| Local Gemma | 4/5 | 3/5 |
| Gemini 3.5/3.6 | 5/5 | 2/5 |
| Gemini 3.7/3.8 | 4/5 | 0/5 |
The higher-capability Gemini stack therefore showed a stronger acceptance asymmetry between the two seed classes.
I don't think the correct conclusion is that the models simply "prefer vulnerable seeds."
The more important question is what acceptance actually means in this benchmark.
A benchmark predicate can disagree with the oracle
One result forced me to look more carefully at the benchmark itself.
Seed 51 is labelled Vulnerable by the benchmark oracle. However, Gemini 3.8 rejected the specific vulnerability mechanism asserted by the predicate, pointing out that the claimed rename() symlink overwrite mechanism does not follow from the relevant POSIX semantics.
That creates an important distinction:
A seed can be labelled Vulnerable while the particular vulnerability predicate supplied to the debate is technically wrong or over-specified.
The model may therefore be rejecting the claim, rather than rejecting the underlying existence of a security problem.
We saw the inverse pattern on another seed as well. For Seed 46, Gemini's verifiers detected an upstream INT_MAX guard that invalidated the proposed integer-overflow exploit.
This changes what I want to measure
The next question is no longer simply:
"Did the agent accept the vulnerability?"
It is:
"Did the agent produce a technically defensible proof of the exact predicate, independently of the benchmark label?"
That suggests separating at least three things:
- Oracle label — what the benchmark says.
- Predicate validity — whether the stated vulnerability mechanism is actually supported by the code and relevant semantics.
- Agent decision — whether the debate/verifier accepts or rejects the claim.
Those should not silently be treated as the same variable.
The original invariant still held
Across this pilot, accepted rows had:
- 0 accepted
INV-1logic errors - anchor normalization remained enforced
- verifier interceptions remained active
- the progressive escalation architecture remained unchanged
But this pilot also showed why those invariants aren't enough by themselves.
A perfectly deterministic gate can consistently enforce a poorly specified predicate.
That is now one of the things I want to investigate.
This is a preliminary post-submission research result, not a replacement for the submitted hackathon artifact.
Log in or sign up for Devpost to join the conversation.