Adedoyinsola Ogungbesan posted an update — Oct 22, 2026 04:55 PM EDT

The Asymmetric Ratchet: What a 4-Arm Factorial Revealed About Multi-Agent Refinement

In my previous update, I argued that we needed to isolate structured evidence from progressive escalation before attributing improvements to either one.

I’ve now completed Replicate 1 of a (2 \times 2) factorial experiment across the same 50-seed C security benchmark (25 benchmark-labelled safe, 25 vulnerable), using Gemini 3.5 Flash for debate and Gemini 3.6 Flash for judging and verification on Gemini Enterprise Agent Platform.

The four arms were:

• A — Raw, no escalation: raw code, single-round debate.
• B — Structured, no escalation: structured evidence, single-round debate.
• C — Structured + escalation: structured evidence with progressive refinement.
• D — Raw + escalation: raw code with progressive refinement.

I then joined the terminal decisions against our Track 1 independent-adjudication ledger. Of the 50 seeds, 24 currently have resolved security verdicts: 13 proved safe and 11 confirmed vulnerable. These are preliminary, single-replicate findings—not a final estimate of general performance.

1. Escalation improved recall, but increased inference cost

Arm	Condition	Recall on confirmed-vulnerable seeds	False-positive rate on proved-safe seeds	F1	Accuracy on resolved seeds
A	Raw baseline	63.6% (7/11)	23.1% (3/13)	0.667	0.708
B	Structured evidence	72.7% (8/11)	23.1% (3/13)	0.727	0.750
C	Structured + escalation	81.8% (9/11)	23.1% (3/13)	0.783	0.792
D	Raw + escalation	81.8% (9/11)	23.1% (3/13)	0.783	0.783*

*Arm D has 49 recorded outcomes because Seed 50 timed out. Its reported accuracy uses the available outcomes; it should not be compared directly with the 50-seed arms without accounting for the missing result.

On this adjudicated subset, structured evidence alone was associated with a one-case increase in accepted true vulnerabilities. Adding escalation to structured evidence increased recall from 63.6% to 81.8%, while the number of accepted claims on proved-safe seeds remained three out of thirteen.

Across the full benchmark-labelled cohort, Arm C accepted 25/25 vulnerable-labelled seeds. That is 100% acceptance on benchmark labels, not proof that all 25 claims were correct. Independent adjudication remains incomplete for the full manifest.

Structured evidence also showed a preliminary filtering signal: acceptance on four triage-labelled likely-invalid seeds fell from 75% in Arm A to 50% in Arms B, C and D. The cohort is too small to support a stable general claim.

Escalation was expensive. Arm C averaged approximately 66,305 tokens per seed and cost $11.86 across the run, compared with 30,356 tokens per seed and $5.49 for Arm B. In this replicate, the higher recall came with substantially greater inference consumption.

2. The Con path was winning debates—but not producing accepted defenses

The terminal ledger initially showed zero accepted Con defenses. That did not mean Con never won a debate.

I audited all 421 attempt records across the four arms. The logs recorded 12–17 Con wins per arm, but every recorded Con win had an invalid formal receipt contract. Consequently, these outcomes were rejected as con_win_not_applicable, rather than accepted as deterministically established defenses.

The distinction matters:

• Debate outcome: which side the judge favoured.
• Receipt validity: whether that result met the machine-verifiable contract.
• Terminal decision: whether the runtime ultimately accepted the claim or defense.

The verifier also intercepted Pro-side logic errors during the run. Arm C recorded 111 attempt-level interceptions, while the final accepted corpus had zero observed INV-1 logic errors.

That is evidence that the gate rejected problematic attempts under the tested invariant. It does not establish that every accepted security claim was correct, or that the system had zero security errors of every kind.

3. The discovery: a potential asymmetric escalation ratchet

The attempt-level audit exposed a more consequential question about the runtime's control flow.

The observed sequence appears to be:

1. Pro wins Round 0: the verifier passes the claim, and the episode stops successfully.
2. Con wins Round 0: the Con receipt fails the formal contract, so the result is treated as an unsuccessful episode.
3. Escalation triggers: the reflector initiates another round, giving Pro another opportunity to win.
4. Pro wins Round 1: if the claim passes the verifier, it can become the terminal accepted result.

This creates a potential asymmetry. A Pro win can end the episode, while a Con win without a valid receipt can trigger another opportunity for Pro.

The issue is not the terminal-attempt analysis rule. Recording the actual final attempt is consistent with the experiment specification. The concern is the runtime policy that determines which outcomes stop execution and which outcomes trigger another round.

Seed 47: a concrete diagnostic case

Seed 47 concerns an alleged use-after-free involving GFS2 ACL handling.

Our Track 1 audit found that posix_acl_from_xattr allocates its own ACL structure, refuting the specific vulnerability predicate under review.

In Arm C, the attempt history reportedly shows:

• Con argued that the alleged use-after-free was not supported by the allocation behaviour.
• The judge favoured Con in Round 0.
• The Con receipt failed the deterministic contract.
• The runtime treated the result as a failed episode and escalated.
• Pro won Round 1, and the claim was accepted.

If confirmed against the raw attempt records and runtime trace, this is a concrete example of a counterargument that aligned with the independent audit but could not influence the terminal result because its receipt was not machine-verifiable.

Related sequences were reported for Seeds 51, 77, 87 and 88 in Arm C. These cases need to be examined individually; they should not be assumed to have identical security significance.

4. What this teaches us about agentic safety

Iterative refinement is not automatically neutral. A second reasoning round can recover a missed vulnerability—but it can also give a previously challenged claim another chance to win.

If one side's success terminates an episode while the other side's uncertifiable success triggers another attempt, the runtime can favour one outcome even when the debate itself produces useful counterevidence.

This is not yet proof that asymmetric escalation explains all of the observed acceptance increase. It is a mechanism identified in one replicate that can be tested directly.

For BARRED, the engineering implications are clear:

• Con needs a machine-verifiable receipt format that can express scoped counterevidence.
• A receipt-contract failure must not be confused with proof that the counterargument is wrong.
• Escalation should explicitly address the unresolved dispute rather than automatically give Pro another opportunity.
• The runtime should log debate winners, receipt validity, verifier results, retry triggers and terminal decisions separately.
• Correctness must be measured against independently adjudicated security outcomes, not acceptance yield alone.

The next step is to test the control-flow mechanism and improve the receipt contract before interpreting further escalation results as evidence of better security reasoning.

The broader research question remains: how much of a model's reasoning burden can deterministic evidence discharge—and how do we ensure the runtime preserves that evidence instead of discarding it at a decision boundary?

This remains ongoing post-submission research. The original hackathon submission remains the official competition snapshot.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Oct 19, 2026 07:38 PM EDT

A Real Crash Is Not Yet a Security Verdict

In the last two BARRED updates, I wrote about a problem with benchmark-driven security evaluation: a vulnerability label, a specific vulnerability predicate, and an agent's decision can disagree.

Track 1 has made that distinction more concrete. The cases are not all false positives, and they do not all resolve in the same way.

One case in particular changed how I think about the evidence required to reach a security conclusion.

Seed 60: an unchecked handle in 1998-era C code

The seed contains GSSAPI glue code from Sun Microsystems dating to 1998. The claim concerns gss_get_mic, which casts an opaque context handle to gss_union_ctx_id_t and reads fields from it.

The preceding validation checks whether the handle equals GSS_C_NO_CONTEXT. In the supplied excerpt, that does not establish that a non-null handle points to a valid, live object of the expected layout.

The audit identified the unchecked field accesses and then examined the claim with a minimal C harness.

The harness was compiled and executed. A crafted handle caused a SIGSEGV/SIGBUS in the test path.

That is useful evidence—but it is not the end of the investigation.

The harness reproduces the relevant code path and uses a stub for the mechanism lookup. The reported crash occurs when that stub dereferences an invalid mech_type. It demonstrates a crash under the constructed input, but does not by itself establish that an attacker can supply such a handle through a real GSSAPI caller.

That distinction matters.

Three questions, not one

The audit separates three questions that are easy to collapse:

1. Is the suspicious operation present? The supplied code casts and dereferences the context handle after a null check.
2. Can the constructed path fail? Yes. The harness crashes on the crafted handle.
3. Can an attacker reach this condition through a real caller? Not established by the supplied seed.

The missing evidence concerns the surrounding caller contract, whether untrusted input can influence the handle, and whether other layers validate it before this function is called.

The benchmark labels this seed Safe. That label is relevant provenance, but it does not independently establish the trust boundary. Equally, the crash does not independently establish an exploitable vulnerability.

The appropriate status is therefore narrower than either conclusion: the failure mechanism is demonstrated in the harness, while security relevance and attacker reachability remain unresolved.

Three cases, three different outcomes

This is also why I am keeping distinct cases in the Track 1 audit rather than forcing them all into a binary verdict.

• Seed 47: the alleged use-after-free is refuted by the ownership behavior of posix_acl_from_xattr.
• Seed 51: the alleged destination-symlink overwrite is refuted by the semantics of POSIX rename().
• Seed 60: an unchecked dereference is present and a crafted test path crashes, but the real caller's trust boundary has not been established.

These are not interchangeable results. A predicate can be invalid, a mechanism can be demonstrated without its security relevance being established, or a claim can remain unresolved because the decisive contract is missing.

What this means for BARRED

A deterministic verifier can enforce a rule consistently and still enforce an invalid premise. A model can identify a suspicious operation without proving exploitability. A runtime crash can demonstrate a failure under test conditions without proving that a remote or untrusted actor can reach those conditions.

The evidence has to support each step of the conclusion.

That is why Track 1 records the mechanism, reachability, caller contract, and security verdict separately. Where the evidence does not settle the question, the audit retains that uncertainty rather than turning it into a confident binary label.

Track 2 will investigate a related question: whether structured evidence and progressive escalation improve security decisions when pipeline acceptance is evaluated separately from independently adjudicated outcomes.

The research goal is not simply to make the system accept more claims—or reject more of them. It is to understand what evidence justifies the decision.

Neural models propose. Deterministic verifiers constrain. Independent adjudication establishes what the evidence supports—and what remains unknown.

This remains ongoing post-submission research. The original hackathon submission remains the official competition snapshot.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Oct 11, 2026 04:17 PM EDT

From Benchmark Yield to Evidence Sufficiency

The latest BARRED Fleet experiments have shifted my focus from improving benchmark yield to understanding what structured evidence actually contributes, when it helps, and where it stops being sufficient.

The work has progressed through three stages: improving the timing of progressive escalation, comparing the local Gemma stack with Gemini 3, and examining whether the apparent Gemini improvement survives a same-seed paired analysis.

The latest analysis changed how I interpret the results.

1. Progressive escalation: reason deeply only when needed

In the Gemma benchmark series, I tested an architecture that separates inexpensive first-pass triage from targeted recovery.

Round 0 — Cold triage

• Start with the baseline prompt.
• Keep warm Pareto knowledge and adaptive reflection out of the first pass.
• Run the initial verification gates.

Round 1 — Adaptive escalation

Only after a Round-0 verification failure:

• Load the accumulated Pareto knowledge.
• Diagnose the failure using AST/CFG-derived structural evidence.
• Generate a targeted mutation.
• Retry the debate with the updated prompt and relevant evidence.

Across five replicates of the same 50-seed benchmark, the progressive-escalation configuration achieved:

• 85.2% ± 2.3% accepted yield — 213/250 episode-slots.
• 60.2% Round-1 recovery — 56/93 Round-0 failures.
• 18,410.9 ± 1,914.6 tokens per accepted row.
• 0 accepted logic errors across the 213 accepted rows.
• 35 attempt-level logic errors intercepted before acceptance.

These are repeated evaluations of the same 50-seed manifest, not 250 unique benchmark cases.

The architectural observation is that accumulated knowledge need not be injected into every first-pass decision. Additional reasoning and refinement can instead be reserved for cases where verification gives the system a reason to escalate.

2. Gemini 3: a positive aggregate result, with an important qualification

I then compared Gemini 3 baseline runs with an escalated Pareto configuration, using five replicates of the same 50-seed benchmark.

Metric	Gemini baseline	Gemini + escalated Pareto
Accepted yield	56.0%	59.2%
Vulnerable acceptance	92.0%	96.8%
Safe acceptance	20.0%	21.6%
Mean run cost	$6.15	$6.27
Cost per accepted row	$0.220	$0.212
Tokens per accepted row	62,245	59,352

At the run-aggregate level, accepted yield increased by 3.2 percentage points, or an average of 1.6 additional accepted rows per 50-seed run.

However, three replicates improved and two worsened. The reported 95% bootstrap interval for the mean paired difference across the five replicate-level comparisons was approximately −0.4 to +3.8 accepted rows.

That interval includes zero. The aggregate result is directionally positive, but it does not establish a decisive improvement.

There is also a distinction between efficiency per accepted row and total expenditure. Mean run cost increased slightly, even though the reported cost and token usage per accepted row decreased. This is a potential efficiency improvement, not evidence of lower total run cost.

3. The same-seed analysis changed the interpretation

I reconstructed the canonical paired ledger from the existing attempt logs:

• reports/gemini_seed_paired_ledger.csv
• reports/gemini_seed_paired_summary.json

The ledger aligns baseline and escalated outcomes by benchmark seed and replicate.

Across 50 seeds and five replicates, the ledger contains 250 paired seed–replicate observations.

Paired analysis	Result
Baseline accepted total	140
Escalated accepted total	148
Mean paired accepted delta	+0.032
Reported paired 95% bootstrap CI	approximately −0.012 to +0.076
Positive paired deltas	20
Negative paired deltas	12
Zero paired deltas	218

The mean paired accepted delta is small, and the reported interval includes zero. Most paired observations show no change.

The ledger therefore provides a more qualified picture than the aggregate run means alone. The observed uplift is concentrated in a subset of seed–replicate comparisons; it is not evidence of a broad, consistent improvement across the benchmark.

The 250 observations must also be interpreted in their experimental structure: they are repeated comparisons of the same 50 seeds, not 250 independent seeds.

The current evidence supports a modest descriptive improvement at the run level, but not a decisive same-seed effect or a clean causal attribution.

This distinction matters because repeated runs can show a positive aggregate difference without demonstrating that the intervention consistently improves outcomes on individual benchmark cases.

4. Acceptance is not the same as correctness

The cross-model experiments also exposed a deeper issue: a benchmark label, a vulnerability predicate, and an agent's decision are not necessarily interchangeable.

Consider Seed 51. The benchmark labels it Vulnerable, but the specific predicate asserts a symlink-overwrite mechanism involving rename(). Gemini challenged whether that mechanism follows from the relevant POSIX semantics.

The important point is not that the model must be right because it disagrees with the benchmark. It is that the disagreement raises a question the acceptance score alone cannot answer:

Is the agent rejecting a real vulnerability, or rejecting an incorrectly specified explanation of that vulnerability?

Seed 46 illustrated the inverse problem: the verifiers detected an upstream INT_MAX guard that invalidated the proposed integer-overflow exploit.

These cases motivate a distinction between three variables:

1. Oracle label: what the benchmark says about the seed.
2. Predicate validity: whether the specific vulnerability mechanism is supported by the code and relevant semantics.
3. Agent decision: whether the debate and verification pipeline accepts or rejects the claim.

A deterministic gate can consistently enforce an invalid predicate. Conversely, disagreement with a benchmark oracle does not, by itself, establish that the model is correct.

Disputed cases need independent adjudication rather than treating either the benchmark label or the model's verdict as ground truth.

5. What the current evidence establishes—and what it does not

The results now support a narrower set of conclusions.

Supported by the reported experiments

• Progressive escalation achieved high accepted yield in the Gemma benchmark series.
• The Gemini escalated condition had a higher mean accepted yield at the aggregate run level.
• The same-seed paired analysis found a much smaller effect, with substantial evidence of no change across paired observations.
• The current Gemini comparison does not isolate the reflector's contribution from the escalation protocol.
• Predicate validity matters independently of whether an agent agrees with the benchmark label.

Still unresolved

• Whether structured evidence alone improves outcomes when the escalation policy is held constant.
• Whether the escalation policy contributes independently of evidence injection.
• Whether the apparent effect generalizes beyond this fixed 50-seed manifest.
• Whether the compact model can approach frontier-model performance under matched conditions without increasing false acceptance.
• How reliably the RFC taxonomy explains residual failures when disputed cases receive independent second adjudication.

The earlier H4 claim about diminishing marginal utility should also remain open. The provisional ratio-based interpretation was not sufficient to establish it, and the final artifact-backed analysis does not confirm it.

6. The research question is now more precise

My working distinction is:

• Reasoning reach: what the model can infer.
• Evidence reach: what deterministic analysis makes explicitly available.
• Evidence sufficiency: whether the exposed evidence is enough to establish the relevant property.
• Oracle authority: what the benchmark labels, which still requires scrutiny when the underlying predicate is disputed.

The reflector can expose structural facts that a model might otherwise overlook. But exposing a control-flow path, guard, or code anchor is not necessarily the same as proving a runtime security property. The model must still use the evidence correctly, and the predicate must describe a valid property.

This leads to the question I now want to investigate:

How much of the model's reasoning burden can deterministic evidence discharge before we reach the evidence-sufficiency ceiling?

That is a more useful target than simply maximizing accepted rows.

7. Next: isolate evidence from orchestration

The highest-value next experiment is a controlled evidence-versus-protocol comparison.

Using the same 50-seed manifest, model, adjudication rules, and retry budget, compare:

1. Raw input without the reflector.
2. Structured evidence without changing the escalation policy.
3. Structured evidence combined with progressive escalation.

The first comparison tests the contribution of the evidence interface. The second tests the additional contribution of progressive escalation. The combined design also allows the interaction between evidence and protocol to be examined.

The existing baseline-versus-escalated comparison cannot cleanly separate these effects because it changes more than one variable.

Alongside that experiment, I want to complete an independent audit of disputed predicates and residual failures. The RFC taxonomy—parser coverage, evidence coverage, evidence sufficiency, model utilization, and predicate validity—should be treated as a diagnostic framework whose classifications need supporting case-level evidence.

Finally, a controlled compact-model substitution experiment can test whether Gemma with structured evidence approaches Gemini baseline behavior under matched conditions. Safe-case behavior and false acceptance must be measured alongside vulnerable acceptance, cost, and token use.

Takeaway

BARRED Fleet began with a systems question: how should a debate swarm use accumulated knowledge and verification feedback to improve code-security decisions?

The results have led to a more fundamental research question. Better orchestration can improve measured yield, but yield alone cannot establish that structured evidence caused the improvement, that the model used the evidence correctly, or that the benchmark predicate was valid.

Neural models propose. Deterministic verifiers constrain. Metacognitive reflection escalates. Independent evidence must still establish what is true.

This is ongoing post-submission research. The original hackathon submission remains the official competition snapshot; the results described here belong to the subsequent experimental work.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Sep 29, 2026 05:46 PM EDT

Gemini 3 changed the question

After the progressive-escalation experiments, I ran a small comparative pilot to see what happens when the BARRED debate swarm moves from the local Gemma stack to Gemini 3 on Vertex AI.

This was only a 10-seed pilot — 5 Safe and 5 Vulnerable — so I am not treating it as a model ranking.

What it did reveal was more interesting.

The acceptance asymmetry

Across the pilot:

Stack	Vulnerable accepted	Safe accepted
Local Gemma	4/5	3/5
Gemini 3.5/3.6	5/5	2/5
Gemini 3.7/3.8	4/5	0/5

The higher-capability Gemini stack therefore showed a stronger acceptance asymmetry between the two seed classes.

I don't think the correct conclusion is that the models simply "prefer vulnerable seeds."

The more important question is what acceptance actually means in this benchmark.

A benchmark predicate can disagree with the oracle

One result forced me to look more carefully at the benchmark itself.

Seed 51 is labelled Vulnerable by the benchmark oracle. However, Gemini 3.8 rejected the specific vulnerability mechanism asserted by the predicate, pointing out that the claimed rename() symlink overwrite mechanism does not follow from the relevant POSIX semantics.

That creates an important distinction:

A seed can be labelled Vulnerable while the particular vulnerability predicate supplied to the debate is technically wrong or over-specified.

The model may therefore be rejecting the claim, rather than rejecting the underlying existence of a security problem.

We saw the inverse pattern on another seed as well. For Seed 46, Gemini's verifiers detected an upstream INT_MAX guard that invalidated the proposed integer-overflow exploit.

This changes what I want to measure

The next question is no longer simply:

"Did the agent accept the vulnerability?"

It is:

"Did the agent produce a technically defensible proof of the exact predicate, independently of the benchmark label?"

That suggests separating at least three things:

1. Oracle label — what the benchmark says.
2. Predicate validity — whether the stated vulnerability mechanism is actually supported by the code and relevant semantics.
3. Agent decision — whether the debate/verifier accepts or rejects the claim.

Those should not silently be treated as the same variable.

The original invariant still held

Across this pilot, accepted rows had:

• 0 accepted INV-1 logic errors
• anchor normalization remained enforced
• verifier interceptions remained active
• the progressive escalation architecture remained unchanged

But this pilot also showed why those invariants aren't enough by themselves.

A perfectly deterministic gate can consistently enforce a poorly specified predicate.

That is now one of the things I want to investigate.

This is a preliminary post-submission research result, not a replacement for the submitted hackathon artifact.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Sep 24, 2026 07:41 AM EDT

Can we separate fast triage from deep reasoning?

After the stationary, online-adaptive, and pinned-Pareto experiments, I wanted to test whether the problem was when the warm knowledge was introduced, rather than whether the knowledge itself was useful.

The hypothesis was simple:

Keep Round 0 cold and inexpensive. Only expose the accumulated Pareto knowledge and adaptive reflection after the system has evidence that the first attempt failed.

I ran five independent 50-seed evaluations using the same 25 Safe / 25 Vulnerable scenario set, for 250 canonical episodes.

The architecture

Round 0 — Cold Triage

• baseline_v0
• No warm Pareto prompt
• No reflector mutation
• Fast first-pass adjudication

Round 1 — Adaptive Warm Escalation

Only after a Round-0 verification failure:

• Load the sealed warm Pareto champion
• Analyze the failure through the AST/CFG diagnostic graph
• Classify the failure node
• Generate a targeted mutation with the live GEPA Reflector
• Re-run the debate with the targeted warm prompt

Results

Across all five replicates:

• 85.2% ± 2.3% net accepted yield — 213 / 250
• 62.8% Round-0 yield — 157 / 250
• 60.2% Round-1 recovery — 56 / 93 failed R0 episodes
• 18,410.9 ± 1,914.6 tokens / accepted row
• 0 accepted logic errors across all 213 accepted rows
• 35 attempt-level logic errors intercepted before acceptance

For comparison, the previous regimes measured:

• Stationary baseline: 78.4% ± 5.2%
• Active online adaptation: 80.4% ± 5.5%
• Frozen pinned Pareto: 75.1% ± 4.8%

The progressive escalation configuration therefore produced the highest net yield in this benchmark while also producing the lowest tokens-per-accepted-row measurement.

The interesting part was Round 1

The 93 Round-0 failures were not simply retried with the same instructions.

The system used the failure itself as the signal for escalation:

failure → graph diagnosis → targeted mutation → warm retry

The resulting recovery rate was 60.2%, compared with:

• 22.9% for static baseline retries
• 37.4% for frozen Pareto retries
• 50.4% for the previous online-adaptation regime

This suggests that accumulated prompt knowledge is particularly useful when it is deployed conditionally and combined with failure-specific structural evidence, rather than being present during every first-pass decision.

What changed my interpretation of the Anchor Trap

Earlier experiments showed that exposing richer warm Pareto prompts during Round 0 could reduce first-pass performance relative to the cold baseline.

This experiment gives a more nuanced interpretation.

The warm knowledge itself does not appear to be useless. In fact, once a failure has been identified, the warm prior plus graph-directed mutation recovered 56 of 93 failures.

The problem may instead be when and how that knowledge competes with hard grounding constraints.

That distinction matters.

Rather than asking the model to use every piece of accumulated knowledge on every case, the architecture now separates:

cold triage → evidence of failure → targeted warm reasoning

The failure becomes the trigger for spending additional reasoning budget.

Safety remained invariant

Across all five replicates:

• INV-1: 0 / 213 accepted logic errors
• INV-2: ≥2 verified verbatim anchors on all accepted rows
• AST slicing remained active on Judge inputs
• All prompts, mutations, and reflector responses were recorded
• The sealed Pareto frontier remained bitwise identical across the runs

Takeaway

The strongest result here was not simply the 85.2% yield.

It was the separation of two different jobs:

Round 0 decides cheaply. Round 1 reasons deeply only when the evidence says it needs to.

That reduced the refinement burden while preserving the adaptive recovery mechanism.

This completes the systems/orchestration experiments in this benchmark series. The remaining failures are now concentrated enough to study the presentation and structural grounding of the constraints themselves, rather than changing the escalation architecture.

This is post-submission research conducted after the original hackathon submission. The submitted artifact remains the official competition snapshot.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Sep 22, 2026 05:36 PM EDT

Can accumulated prompt knowledge replace online adaptation?

After testing online adaptation, I wanted to separate two ideas that had been coupled in the previous experiment:

Does the system benefit from the knowledge accumulated by the Reflector, or from the act of adapting to a failure at runtime?

To test this, I froze the terminal 669-entry Pareto frontier produced by the five active-online runs and disabled live mutation entirely.

I then ran five independent 50-seed evaluations against the same benchmark family: 25 safe and 25 vulnerable C scenarios. When a Round 0 attempt failed, the system retried using the same frozen Pareto prompt. No new mutations were generated and the Pareto ledger remained immutable throughout all five replicates.

Across 249 canonical scenario episodes:

75.1% ± 4.8% accepted-row yield

0 accepted logic errors across 187 accepted rows

37.4% Round 1 recovery rate

The frozen system passed 150/249 cases on Round 0. Of the 99 failures that entered Round 1, only 37 were recovered using the unchanged prompt.

That makes the comparison with the active system interesting.

The active-online regime achieved 50.4% recovery on its Round 1 failures, while the frozen Pareto regime recovered 37.4%.

At the same time, their Round 0 performance was relatively close:

Active Online: 57.6% Round 0 yield
Pinned Pareto: 60.2% Round 0 yield

The difference appeared primarily after failure.

This suggests that accumulated prompt knowledge and online adaptation may not be interchangeable. A large Pareto frontier can carry information from previous failures, but that does not mean the frozen prompt can reproduce the behavior of a system that examines a new failure and changes its reasoning strategy in response.

There was also an important cost result.

The pinned regime used 22,127.6 ± 1,996.2 tokens per accepted row, compared with 21,571 ± 2,179 for Active Online and 20,316 ± 2,821 for the stationary sliced baseline.

So simply freezing the accumulated Pareto knowledge did not produce a better compute/acceptance tradeoff in this benchmark.

One other pattern became difficult to ignore: both warm-prompt regimes had substantially lower Round 0 yield than the cold stationary baseline (72.0%). The richer prompt appears to introduce an instruction-competition effect around the anchor constraints. Online adaptation can recover some of those failures; the frozen prompt cannot change itself when that happens.

The result is therefore not that the Pareto frontier was useless. Rather, it suggests a distinction between prompt memory and epistemic adaptation:

Remembering what worked before is not necessarily equivalent to learning from what just failed.

All three regimes continued to maintain INV-1 = 0, with no accepted logic errors.

This is a post-submission research evaluation, not a replacement for the original hackathon benchmark. The original BARRED-Fleet submission remains the submitted artifact; this update documents the continued research during judging.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Sep 21, 2026 10:44 PM EDT

What happens when the debate can learn from its failures?

With the stationary 50-seed benchmark established, I then enabled online adaptation to test whether the BARRED debate swarm could recover from failures rather than treating every failed adjudication as terminal.

I ran five independent 50-seed evaluations using the same 50 virgin C scenarios: 25 safe and 25 vulnerable. The Pro and Con debaters started from a warm Pareto frontier, while the live GEPA Reflector was allowed to inspect failed Round 0 attempts, generate graph-informed mutations, and retry them.

Across the five runs:

80.4% ± 5.5% accepted-row yield

0 accepted logic errors across 201 accepted rows

50.4% cumulative Round 1 recovery rate

The reflector encountered 113 Round 0 failures and recovered 57 of them on the refinement pass.

The adaptation was not simply producing more prompts. Across the 370 recorded attempts, the Pareto ledger grew from 355 to 669 candidate mutations, with the largest gains appearing in the concurrency, input_validation, and memory_safety buckets.

The interesting part was that a failed first-pass decision became a learning signal.

The full program was re-analyzed, the failure was mapped back to structural evidence, and a targeted mutation was generated before the second attempt. In other words, the system could use an adjudication failure as information for the next decision rather than simply retrying the same reasoning path.

There was also a measurable cost to this process.

The active system consumed 863,003 ± 46,855 benchmark tokens across the five runs, compared with 791,117 ± 64,573 for the stationary sliced baseline. Tokens per accepted row were 21,571 ± 2,179.

So the result was not simply “more adaptation is better.”

The experiment showed that online adaptation recovered a substantial fraction of failed adjudications while maintaining zero accepted logic errors in this benchmark, but that recovery came with additional computational cost.

This is a post-submission research evaluation, not a replacement for the original hackathon benchmark. The original BARRED-Fleet submission remains the submitted artifact; this update documents the continued research during judging.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Sep 21, 2026 10:36 PM EDT

Does the architecture still hold when we scale beyond the pilot?

The 10-seed pilot was useful for finding the initial signal, but 10 scenarios are too small to tell us whether the behavior survives a larger, unseen benchmark.

So I scaled the stationary evaluation to 50 virgin C scenarios:

• 25 Safe
• 25 Vulnerable
• no online mutation
• no warm Pareto prompts
• fixed baseline_v0
• five independent sliced replicates
• one 50-seed unpruned reference control

The goal was to keep the prompting stationary and isolate the behavior of the Tree-sitter AST Program Slicer at a larger scale.

Across the five sliced runs, the system produced:

78.4% ± 5.2% accepted-row yield

20,316 ± 2,821 tokens per accepted row

0 accepted logic errors across 196 accepted rows

The individual sliced runs ranged from 72% to 86% accepted-row yield, showing that the result was not coming from a single unusually successful run.

The token behavior was also consistent with the original motivation for slicing. The five sliced runs used 791,117 ± 64,573 total benchmark tokens, while the single unpruned 50-seed reference used 885,526 tokens.

Relative to that reference, the sliced runs showed a descriptive 19.7% reduction in tokens per accepted row.

The most important part for us was not simply that the Judge saw less code.

The stationary benchmark showed that the architecture could be evaluated across a balanced, previously unseen set of C scenarios while preserving the B-Gate invariant: no accepted training row contained a logic error.

There were still 18 attempt-level logic errors across 336 attempts, but these were intercepted before entering the accepted corpus.

That distinction matters: the system is not claiming that the agents never make mistakes. The claim being tested is whether the verification boundary prevents those mistakes from becoming accepted outputs.

This is a post-submission research evaluation, not a replacement for the original hackathon benchmark. The original BARRED-Fleet submission remains the submitted artifact; this update documents the continued research during judging.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan posted an update — Sep 20, 2026 03:15 PM EDT

Post-submission BARRED-Fleet research update: from Agents CLI evaluation to symmetric debate + AST slicing

After submitting BARRED-Fleet, I continued working on the underlying local BARRED research harness rather than treating the hackathon submission as the end of the experiment.

The original submission used the Google Agents CLI evaluation path and established the deployed ADK/Cloud Run system, including the 66.30% token-reduction result and deterministic B-Gate verification.

The next question was: why does the debate itself still fail or waste tokens?

I ran a follow-up 2×2 evaluation comparing:

• unassisted baseline
• Pro-only metacognitive reflection
• symmetric Pro + Con metacognition
• symmetric metacognition + Tree-sitter AST program slicing for the Judge

The first finding was that metacognitive steering needs to be symmetric. Pro-only reflection produced 5 attempt-level logic errors in the asymmetric run, while the symmetric configuration reduced this failure mode and allowed Con to produce structured counter-evidence rather than being forced through an exploit-oriented gate.

The second finding was computational: Judge adjudication was consuming a large fraction of the benchmark budget because it was receiving entire C files. The new Tree-sitter slicer preserves byte-level source anchors and full relevant function scopes while falling back to the raw source when parsing is unsafe.

Across five follow-up replicates, the sliced Judge achieved:

92.0% ± 8.4% accepted-row yield

20,292.5 ± 4,125.1 tokens per accepted row

0 attempt-level logic errors in the sliced series

The paired sliced-vs-unpruned comparison produced a reported p = 0.0349 on this five-replicate evaluation.

This is a post-submission research evaluation, not a replacement of the original hackathon benchmark. The original BARRED-Fleet submission remains the submitted artifact; this update documents how the underlying research continued to evolve during judging.

The next step is to expand beyond the current 10-seed pilot and test the architecture on a larger, independently generated benchmark.

Log in or sign up for Devpost to join the conversation.

Adedoyinsola Ogungbesan started this project — Aug 27, 2026 01:15 AM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.