Does the architecture still hold when we scale beyond the pilot?
The 10-seed pilot was useful for finding the initial signal, but 10 scenarios are too small to tell us whether the behavior survives a larger, unseen benchmark.
So I scaled the stationary evaluation to 50 virgin C scenarios:
- 25 Safe
- 25 Vulnerable
- no online mutation
- no warm Pareto prompts
- fixed
baseline_v0 - five independent sliced replicates
- one 50-seed unpruned reference control
The goal was to keep the prompting stationary and isolate the behavior of the Tree-sitter AST Program Slicer at a larger scale.
Across the five sliced runs, the system produced:
78.4% ± 5.2% accepted-row yield
20,316 ± 2,821 tokens per accepted row
0 accepted logic errors across 196 accepted rows
The individual sliced runs ranged from 72% to 86% accepted-row yield, showing that the result was not coming from a single unusually successful run.
The token behavior was also consistent with the original motivation for slicing. The five sliced runs used 791,117 ± 64,573 total benchmark tokens, while the single unpruned 50-seed reference used 885,526 tokens.
Relative to that reference, the sliced runs showed a descriptive 19.7% reduction in tokens per accepted row.
The most important part for us was not simply that the Judge saw less code.
The stationary benchmark showed that the architecture could be evaluated across a balanced, previously unseen set of C scenarios while preserving the B-Gate invariant: no accepted training row contained a logic error.
There were still 18 attempt-level logic errors across 336 attempts, but these were intercepted before entering the accepted corpus.
That distinction matters: the system is not claiming that the agents never make mistakes. The claim being tested is whether the verification boundary prevents those mistakes from becoming accepted outputs.
This is a post-submission research evaluation, not a replacement for the original hackathon benchmark. The original BARRED-Fleet submission remains the submitted artifact; this update documents the continued research during judging.
Log in or sign up for Devpost to join the conversation.