Fencepost

Inspiration

I have friends in CS courses and they always talk about the same thing. Now an AI agent can do the whole homework. So a student sends the code, all tests pass, green everywhere. But did they understand it? Or did the agent do it while they watched?

The teachers know some students used an agent. But they can't prove who. So they grade like nobody did it. I did not want to make another AI detector, those don't work and they hurt honest students. So I thought, ok, different question: does this student understand their own code? That one you can answer.

What it does

Fencepost takes a line the student wrote (Git tells us it's theirs), and changes one character. Just one. Then it runs the student's own tests on that change. If their tests don't notice, that's something they never really checked.

Then GPT-5.6 writes a test that shows the difference, a sandbox runs it, and the failing test is the answer. The teacher gets a short report with a few real questions to ask. No grade, no verdict. It's a reason to talk to the student.

How I built it

I used Codex with GPT-5.6 for the engine. I wrote the spec, made the design, and I walked the product myself many times to see what was confusing. Codex is also inside the product: it runs GPT-5.6 at the moment, on the teacher's own plan, no API key, in a Docker sandbox with no internet.

I made the demo student fake on purpose, with the real answers written down, so anyone can check every number without an API key.

Challenges

The hardest part was the "equivalent" changes. Sometimes you change a line but it does exactly the same, so asking about it is not fair. I thought I found one and I was wrong. GPT-5.6 proved me wrong two times. That taught me something: "equal" only means something if you say the rules first. So the tool shows two numbers, never just one "truth".

The design was also hard. My first UI was ugly and I said it, and I kept pushing until it felt right. Same with the video.

What I learned

I learned mutation testing, how to change code with AST, and how to run untrusted code safe in a sandbox. But the big lesson was about being honest. A human and two AI models were all sure, and all wrong. Only running the test showed the truth. That's exactly what Fencepost does to a student's green tests.

What's next

One real CS2 class this fall. It's MIT license and the report stays local.

Built With

• ast
• codex
• docker
• education
• git
• gpt-5.6
• mutation-testing
• pytest
• python
• remotion