Adedoyinsola Ogungbesan posted an update — Oct 11, 2026 04:17 PM EDT

From Benchmark Yield to Evidence Sufficiency

The latest BARRED Fleet experiments have shifted my focus from improving benchmark yield to understanding what structured evidence actually contributes, when it helps, and where it stops being sufficient.

The work has progressed through three stages: improving the timing of progressive escalation, comparing the local Gemma stack with Gemini 3, and examining whether the apparent Gemini improvement survives a same-seed paired analysis.

The latest analysis changed how I interpret the results.

1. Progressive escalation: reason deeply only when needed

In the Gemma benchmark series, I tested an architecture that separates inexpensive first-pass triage from targeted recovery.

Round 0 — Cold triage

• Start with the baseline prompt.
• Keep warm Pareto knowledge and adaptive reflection out of the first pass.
• Run the initial verification gates.

Round 1 — Adaptive escalation

Only after a Round-0 verification failure:

• Load the accumulated Pareto knowledge.
• Diagnose the failure using AST/CFG-derived structural evidence.
• Generate a targeted mutation.
• Retry the debate with the updated prompt and relevant evidence.

Across five replicates of the same 50-seed benchmark, the progressive-escalation configuration achieved:

• 85.2% ± 2.3% accepted yield — 213/250 episode-slots.
• 60.2% Round-1 recovery — 56/93 Round-0 failures.
• 18,410.9 ± 1,914.6 tokens per accepted row.
• 0 accepted logic errors across the 213 accepted rows.
• 35 attempt-level logic errors intercepted before acceptance.

These are repeated evaluations of the same 50-seed manifest, not 250 unique benchmark cases.

The architectural observation is that accumulated knowledge need not be injected into every first-pass decision. Additional reasoning and refinement can instead be reserved for cases where verification gives the system a reason to escalate.

2. Gemini 3: a positive aggregate result, with an important qualification

I then compared Gemini 3 baseline runs with an escalated Pareto configuration, using five replicates of the same 50-seed benchmark.

Metric	Gemini baseline	Gemini + escalated Pareto
Accepted yield	56.0%	59.2%
Vulnerable acceptance	92.0%	96.8%
Safe acceptance	20.0%	21.6%
Mean run cost	$6.15	$6.27
Cost per accepted row	$0.220	$0.212
Tokens per accepted row	62,245	59,352

At the run-aggregate level, accepted yield increased by 3.2 percentage points, or an average of 1.6 additional accepted rows per 50-seed run.

However, three replicates improved and two worsened. The reported 95% bootstrap interval for the mean paired difference across the five replicate-level comparisons was approximately −0.4 to +3.8 accepted rows.

That interval includes zero. The aggregate result is directionally positive, but it does not establish a decisive improvement.

There is also a distinction between efficiency per accepted row and total expenditure. Mean run cost increased slightly, even though the reported cost and token usage per accepted row decreased. This is a potential efficiency improvement, not evidence of lower total run cost.

3. The same-seed analysis changed the interpretation

I reconstructed the canonical paired ledger from the existing attempt logs:

• reports/gemini_seed_paired_ledger.csv
• reports/gemini_seed_paired_summary.json

The ledger aligns baseline and escalated outcomes by benchmark seed and replicate.

Across 50 seeds and five replicates, the ledger contains 250 paired seed–replicate observations.

Paired analysis	Result
Baseline accepted total	140
Escalated accepted total	148
Mean paired accepted delta	+0.032
Reported paired 95% bootstrap CI	approximately −0.012 to +0.076
Positive paired deltas	20
Negative paired deltas	12
Zero paired deltas	218

The mean paired accepted delta is small, and the reported interval includes zero. Most paired observations show no change.

The ledger therefore provides a more qualified picture than the aggregate run means alone. The observed uplift is concentrated in a subset of seed–replicate comparisons; it is not evidence of a broad, consistent improvement across the benchmark.

The 250 observations must also be interpreted in their experimental structure: they are repeated comparisons of the same 50 seeds, not 250 independent seeds.

The current evidence supports a modest descriptive improvement at the run level, but not a decisive same-seed effect or a clean causal attribution.

This distinction matters because repeated runs can show a positive aggregate difference without demonstrating that the intervention consistently improves outcomes on individual benchmark cases.

4. Acceptance is not the same as correctness

The cross-model experiments also exposed a deeper issue: a benchmark label, a vulnerability predicate, and an agent's decision are not necessarily interchangeable.

Consider Seed 51. The benchmark labels it Vulnerable, but the specific predicate asserts a symlink-overwrite mechanism involving rename(). Gemini challenged whether that mechanism follows from the relevant POSIX semantics.

The important point is not that the model must be right because it disagrees with the benchmark. It is that the disagreement raises a question the acceptance score alone cannot answer:

Is the agent rejecting a real vulnerability, or rejecting an incorrectly specified explanation of that vulnerability?

Seed 46 illustrated the inverse problem: the verifiers detected an upstream INT_MAX guard that invalidated the proposed integer-overflow exploit.

These cases motivate a distinction between three variables:

1. Oracle label: what the benchmark says about the seed.
2. Predicate validity: whether the specific vulnerability mechanism is supported by the code and relevant semantics.
3. Agent decision: whether the debate and verification pipeline accepts or rejects the claim.

A deterministic gate can consistently enforce an invalid predicate. Conversely, disagreement with a benchmark oracle does not, by itself, establish that the model is correct.

Disputed cases need independent adjudication rather than treating either the benchmark label or the model's verdict as ground truth.

5. What the current evidence establishes—and what it does not

The results now support a narrower set of conclusions.

Supported by the reported experiments

• Progressive escalation achieved high accepted yield in the Gemma benchmark series.
• The Gemini escalated condition had a higher mean accepted yield at the aggregate run level.
• The same-seed paired analysis found a much smaller effect, with substantial evidence of no change across paired observations.
• The current Gemini comparison does not isolate the reflector's contribution from the escalation protocol.
• Predicate validity matters independently of whether an agent agrees with the benchmark label.

Still unresolved

• Whether structured evidence alone improves outcomes when the escalation policy is held constant.
• Whether the escalation policy contributes independently of evidence injection.
• Whether the apparent effect generalizes beyond this fixed 50-seed manifest.
• Whether the compact model can approach frontier-model performance under matched conditions without increasing false acceptance.
• How reliably the RFC taxonomy explains residual failures when disputed cases receive independent second adjudication.

The earlier H4 claim about diminishing marginal utility should also remain open. The provisional ratio-based interpretation was not sufficient to establish it, and the final artifact-backed analysis does not confirm it.

6. The research question is now more precise

My working distinction is:

• Reasoning reach: what the model can infer.
• Evidence reach: what deterministic analysis makes explicitly available.
• Evidence sufficiency: whether the exposed evidence is enough to establish the relevant property.
• Oracle authority: what the benchmark labels, which still requires scrutiny when the underlying predicate is disputed.

The reflector can expose structural facts that a model might otherwise overlook. But exposing a control-flow path, guard, or code anchor is not necessarily the same as proving a runtime security property. The model must still use the evidence correctly, and the predicate must describe a valid property.

This leads to the question I now want to investigate:

How much of the model's reasoning burden can deterministic evidence discharge before we reach the evidence-sufficiency ceiling?

That is a more useful target than simply maximizing accepted rows.

7. Next: isolate evidence from orchestration

The highest-value next experiment is a controlled evidence-versus-protocol comparison.

Using the same 50-seed manifest, model, adjudication rules, and retry budget, compare:

1. Raw input without the reflector.
2. Structured evidence without changing the escalation policy.
3. Structured evidence combined with progressive escalation.

The first comparison tests the contribution of the evidence interface. The second tests the additional contribution of progressive escalation. The combined design also allows the interaction between evidence and protocol to be examined.

The existing baseline-versus-escalated comparison cannot cleanly separate these effects because it changes more than one variable.

Alongside that experiment, I want to complete an independent audit of disputed predicates and residual failures. The RFC taxonomy—parser coverage, evidence coverage, evidence sufficiency, model utilization, and predicate validity—should be treated as a diagnostic framework whose classifications need supporting case-level evidence.

Finally, a controlled compact-model substitution experiment can test whether Gemma with structured evidence approaches Gemini baseline behavior under matched conditions. Safe-case behavior and false acceptance must be measured alongside vulnerable acceptance, cost, and token use.

Takeaway

BARRED Fleet began with a systems question: how should a debate swarm use accumulated knowledge and verification feedback to improve code-security decisions?

The results have led to a more fundamental research question. Better orchestration can improve measured yield, but yield alone cannot establish that structured evidence caused the improvement, that the model used the evidence correctly, or that the benchmark predicate was valid.

Neural models propose. Deterministic verifiers constrain. Metacognitive reflection escalates. Independent evidence must still establish what is true.

This is ongoing post-submission research. The original hackathon submission remains the official competition snapshot; the results described here belong to the subsequent experimental work.

Log in or sign up for Devpost to join the conversation.