Viva

GPT-5.6 finds the input a test suite never tried, runs the student's code against the instructor's, and asks them to explain the one result they can't fake.

Comment

Inspiration

A test suite only checks the inputs somebody thought to write down. If a student's code happens to work on those, it gets full marks, and nobody ever finds out whether they could explain it.

I wanted to know how often that actually matters in real student code rather than in theory. So I took a public dataset of intro-programming submissions, kept only the ones that scored full marks, and started running them on inputs their graders never used. The first exercise I opened asks for the largest of three integers. One submission scored 3/3 and prints 3 when you give it 5 5 3, because it chains strict greater-than comparisons and a tie falls through to the final else. The instructor's three tests never tie, so nothing caught it.

That is the whole idea. The grade was measuring the test suite, not the student.

What it does

Viva takes a student's C submission and the instructor's own reference solution.

GPT-5.6 reads the assignment specification and the reference, then proposes inputs the test suite most likely never tried: ties, zero, negatives, single elements, boundaries. Viva compiles and runs both programs on every proposed input and only opens a code defense when the two disagree on an input that reproduces on a second run.

The student then sees the exact input and both outputs and explains, in their own words, why their program did that. GPT-5.6 grades only that explanation against the executed evidence. If the student restates the output instead of tracing the code, it pushes back with a sharper question. It never asks whether they used AI. It asks what they can explain.

The teacher gets a brief: which misconceptions survived the test suite, ranked by how many full-mark students they caught, and a twenty-minute reteach plan built from the exact input that exposed the gap.

On 626 submissions that all earned full marks, across three exercises, Viva found 91 (14.5%) that hide a defect the autograder never checked. All 215 findings replayed on an independent rerun. The dominant misconception is ties: 83 of those students print the wrong answer on a case as simple as 5 5 3.

How I built it

The design splits one job into two very different ones.

GPT-5.6 decides what to test. It reads the spec and the instructor's reference and proposes spec-aware edge cases, which is the part that generalizes: it produced discovery sets for all 25 exercises in the corpus, including the float, digit-sum and time-conversion problems where a mechanical mutation of the existing tests produces nonsense.

Execution decides whether this submission actually fails. A runner compiles the student program and the reference with GCC (through WSL on Windows) and runs every proposed input on both. A finding only survives if it reproduces. That means a probe is never the model's opinion about the code, it is two programs printing different things on the same input, with both outputs shown.

GPT-5.6 then conducts the defense, constrained to that one probe and checked for the expected evidence id before its response reaches the interface.

I built it in Codex with GPT-5.6. Codex accelerated the parts that were fiddly rather than clever: the WSL C runner, batching every candidate input for a submission into a single call so the corpus run finishes in minutes instead of hours, the evidence schema, the divergence categorizer, the benchmark harness, and the deployed teacher and student pages.

Challenges I ran into

Keeping the benchmark honest was harder than building it.

My first corpus run reported a much larger defect rate, and it was wrong. Mutating the instructor's test inputs without understanding the spec produces out-of-spec inputs: feeding a count of zero to an averaging program made the reference print -nan, and the student disagreeing with that is not evidence of anything. Exercises whose spec says "positive integers" were being fed zero and negatives. A string-processing exercise was getting inputs labelled with concepts that made no sense for it.

So I cut the number down. The reported benchmark now only scores exercises where every proposed input is genuinely in-domain and every concept label is correct. I added a guard that ignores any input where the reference itself prints nan or inf, and a categorizer that separates real answer-changing divergences from cosmetic float formatting, so a 5.0 versus 5.000000 difference is detected and excluded rather than counted. The result is smaller and defensible: 91 findings, all 215 replayed, zero cosmetic false positives.

The other good one was a crash. A student program stuck in an infinite print loop flooded stdout fast enough to overflow the string buffer before the timeout could kill it, and took the whole corpus run down with it. Captured output is now capped, so one runaway submission can't end the run.

Accomplishments that I'm proud of

Every question starts from a replayed execution difference. There is no step where the model is asked what it thinks of the code, so there is no step where it can invent a bug that isn't there.

I'm more proud of the number I didn't report. It would have been easy to keep the bigger defect rate and never mention that some of those inputs were out of spec. Cutting it to the set I can defend line by line is the part I'd want a teacher to check.

And it never makes an authorship claim. An explanation gap is a teaching signal, not an accusation.

What I learned

The model is most useful when it is pointed at the one thing it is genuinely better at than a script: reading a specification and imagining how a beginner would get it wrong. Everything downstream of that is better handled by actually running the program.

I also learned how much of a benchmark's credibility lives in what you exclude. The categorizer and the in-domain restriction took longer than the discovery engine and matter more.

What's next for Viva

A consented classroom pilot with educator review, then LMS integration so the brief arrives where the teacher already works. The discovery engine already covers all 25 exercises in the corpus; the next step is validating those findings the same way, then testing whether the follow-up questions actually help students repair the misconception rather than just exposing it.

Built With

Share this project:

Updates