Transfer: proof you actually learned it

An AI that refuses to give you the answer. It diagnoses what you misunderstood, then proves you learned it by making you solve a problem you have never seen, unaided.

Comment

The thing that bothered me

A friend was working through a Python course and getting every exercise right. Every single one. Then I asked him to write something slightly different and he froze. He had not learned Python. He had learned how to ask for answers.

That is the problem in one sentence. Every student now has a model that will hand them working code. Tests go green, the progress bar fills, and nobody, including the student, knows whether anything stuck.

Then a second thing bothered me more, because it has nothing to do with AI. Passing a problem you just practiced has never proved much. You might have pattern matched your way through it. Learning science has a name for the real thing: transfer. Can you use the idea in a situation you have not seen before? Almost no software measures that, because measuring it is hard. Generating content is easy. Verifying understanding is not.

So I built the thing that measures it.

What it does

Transfer teaches fifteen Python concepts, and it has one rule that makes it different from every practice site: it will not give you the answer, and it will not call anything mastered until you prove it on a problem you have never seen.

The loop:

  1. You run code. It executes for real in a Python runtime in your browser.
  2. When it fails, GPT-5.6 reads your actual code and names the misconception behind the mistake. Not "wrong," but "you believe range includes its stop value."
  3. If you ask for help you get a Socratic question, never a solution. A server side guard inspects every hint and rejects anything that leaks an answer, including hints the model itself generated.
  4. Once you fix the original, the app generates a brand new isomorphic problem. Same concept, different surface. It comes with its own reference solution and test cases, and the app runs that reference solution against those tests before you ever see the challenge, so you are never handed a broken problem.
  5. Hints lock. You solve the new one alone. Your code is graded by execution, not by an opinion.
  6. Only then does the concept turn to mastered, and you get a checksummed evidence record showing the failed attempt, the diagnosis, the generated problem, and the unaided solve.

Then there is the part I did not expect to care about most. Scale it to a classroom and you get a signal no gradebook produces: who passed the practice problem and then failed the transfer. That is fragile understanding, visible before the exam instead of after it. The teacher view ranks those signals per concept and suggests what to reteach.

How I built it with Codex and GPT-5.6

I wrote a VISION file first, then drove Codex on GPT-5.6 through the whole build, reviewing each product decision instead of accepting output.

Codex wrote the application: the Pyodide execution sandbox and its web worker, the mastery state machine, the answer refusal guard, the self verifying transfer engine, the tamper evident proof records, and the cohort analytics. It also wrote the test suite, which grew to 99 passing tests covering the mastery invariants, answer refusal, proof integrity, and live mode fallback.

GPT-5.6 did something beyond writing code: it authored the teaching content itself. The misconceptions for each concept, the three level hint trees, and fifteen concepts worth of transfer problems with their test harnesses. That content ships in the repo as a verified bank.

At runtime the app uses live GPT-5.6 for diagnosis, hint wording, and fresh challenge generation. Every live response is schema validated, timed out, and passed through the refusal guard. If anything fails it falls back to the verified bank, so the learner never hits a dead end. With no API key at all the app still runs completely, because execution and transfer verification were never the model's job.

The division that made this work: the model generates, the runtime verifies. Mastery is decided by running code, not by asking a model whether the learner deserves it.

What was hard

Making an AI refuse to help. Models want to be useful. Asked for a hint about an off by one error, GPT-5.6 would happily write the corrected line. The fix was a guard that treats model output as untrusted and rejects anything code shaped, plus prompts that only permit questions. Tuning it was fiddly, because early versions also rejected perfectly good Socratic hints that happened to contain the word "change."

Trusting a generated problem. A transfer challenge is worthless if its tests are wrong, and worse than worthless if it marks a correct learner as failing. So generation produces the problem, a reference solution, and the tests together, and the app executes the reference solution against those tests before presenting anything. If it fails, it regenerates.

Defining mastery so it cannot be gamed. My first state machine granted mastery on any passing transfer attempt, including one where hints had been used. That quietly broke the entire premise. Mastery now requires zero hints during the transfer phase, enforced in the state machine rather than in the interface, with a regression test that fails if anyone loosens it.

What I learned

The hard part of AI in education is not generation. It is verification. Once I stopped asking the model to judge learning and started making the runtime prove it by executing code, the product got simpler and much more trustworthy at the same time.

Being straight about the limits

The cohort dashboard uses clearly labelled seeded demo data. There is no roster integration or live classroom.

The proof record is an integrity check. It can show that a record was not altered and that its steps are self consistent. It is not identity verification, not a signature, and not an anti cheating system. It records integrity signals such as pastes during the transfer phase, it does not prevent them.

It covers Python fundamentals. Fifteen concepts, not a curriculum.

What is next

More subjects, since the loop is not Python specific. Durable learner records. And the piece I most want to test: whether an employer or admissions office would treat an execution backed transfer rec

Built With

Share this project:

Updates