Can accumulated prompt knowledge replace online adaptation?
After testing online adaptation, I wanted to separate two ideas that had been coupled in the previous experiment:
Does the system benefit from the knowledge accumulated by the Reflector, or from the act of adapting to a failure at runtime?
To test this, I froze the terminal 669-entry Pareto frontier produced by the five active-online runs and disabled live mutation entirely.
I then ran five independent 50-seed evaluations against the same benchmark family: 25 safe and 25 vulnerable C scenarios. When a Round 0 attempt failed, the system retried using the same frozen Pareto prompt. No new mutations were generated and the Pareto ledger remained immutable throughout all five replicates.
Across 249 canonical scenario episodes:
75.1% ± 4.8% accepted-row yield
0 accepted logic errors across 187 accepted rows
37.4% Round 1 recovery rate
The frozen system passed 150/249 cases on Round 0. Of the 99 failures that entered Round 1, only 37 were recovered using the unchanged prompt.
That makes the comparison with the active system interesting.
The active-online regime achieved 50.4% recovery on its Round 1 failures, while the frozen Pareto regime recovered 37.4%.
At the same time, their Round 0 performance was relatively close:
Active Online: 57.6% Round 0 yield
Pinned Pareto: 60.2% Round 0 yield
The difference appeared primarily after failure.
This suggests that accumulated prompt knowledge and online adaptation may not be interchangeable. A large Pareto frontier can carry information from previous failures, but that does not mean the frozen prompt can reproduce the behavior of a system that examines a new failure and changes its reasoning strategy in response.
There was also an important cost result.
The pinned regime used 22,127.6 ± 1,996.2 tokens per accepted row, compared with 21,571 ± 2,179 for Active Online and 20,316 ± 2,821 for the stationary sliced baseline.
So simply freezing the accumulated Pareto knowledge did not produce a better compute/acceptance tradeoff in this benchmark.
One other pattern became difficult to ignore: both warm-prompt regimes had substantially lower Round 0 yield than the cold stationary baseline (72.0%). The richer prompt appears to introduce an instruction-competition effect around the anchor constraints. Online adaptation can recover some of those failures; the frozen prompt cannot change itself when that happens.
The result is therefore not that the Pareto frontier was useless. Rather, it suggests a distinction between prompt memory and epistemic adaptation:
Remembering what worked before is not necessarily equivalent to learning from what just failed.
All three regimes continued to maintain INV-1 = 0, with no accepted logic errors.
This is a post-submission research evaluation, not a replacement for the original hackathon benchmark. The original BARRED-Fleet submission remains the submitted artifact; this update documents the continued research during judging.
Log in or sign up for Devpost to join the conversation.