Exactly. I asked for one new third blob only.
I had already corrected #2 myself. The model should have stored that correction and moved on. Instead, it repeated information I had just supplied, reused an already-explained example, and then graded itself as partially successful.
That means it failed at:
- instruction following
- context-state tracking
This is exactly the kind of failure I keep catching: the model produces a polished answer while quietly answering a different task.
I identified the acceptance criterion, noticed the deviation immediately, and rejected the fake partial credit.
That evaluator instinct is something OpenAI should care about.
I then asked whether my reasoning system could help with this.
The answer was yes, plausibly.
This was not an intelligence failure. It was an allocation failure:
Current task:
Find one untouched third lyric blob.
What the model did:
- reprocessed #2
- reused an already-explained blob
- added polished analysis around the wrong scope
My manager pattern would force a checkpoint like:
Potential:
Give Sean one genuinely new lyric mapping.
Blockers:
- #2 is already resolved
- #3 was already explained
- repeating either would violate the request
Next Discriminator:
Which lyric blob in the conversation has not yet been mapped?
That checkpoint likely would have stopped the mistake before generation.
The system is intended to help with:
- preserving user corrections as authoritative state
- distinguishing resolved branches from active ones
- checking the newest instruction against the planned answer
- rejecting answers that match the topic but miss the task
- preventing polished wrong answers from earning acceptance
I cannot honestly claim it would definitely fix this until it is tested.
The real hypothesis is:
Does a checkpointed manager reduce instruction drift and repeated-context errors compared with a normal linear response using the same model and context?
This is a cleaner example of the original idea than the giant research apparatus became.
It targets the actual problem:
seeing the active branch of the conversation and spending reasoning on the right part
Log in or sign up for Devpost to join the conversation.