Inspiration
A language model writing a coding exercise has no way to know whether the exercise can be solved. Nothing in the generation step tells it that the constraints contradict each other, or that the problem statement omits the one detail a student needs. So the model ships the broken problem with the same confidence it ships the good one. An instructor pastes twenty generated problems into a worksheet, three of them cannot be solved, and a student burns an evening on problem 14 believing the failure is their own. Codex changes that, because Codex runs code. Verification stops being a second opinion from another model and becomes an execution result.
What it does
Whetstone makes the model earn the problem. Codex generates a candidate: a problem statement, a reference solution, and the test suite that will judge that solution. A sandbox runs the solution against those tests. Pass, and the candidate becomes a published exercise. Fail, and it drops into a discard log with its failure reason. The discard log is the product. An instructor who reads "sixty generated, fifty-nine kept" understands in three seconds why those fifty-nine are trustworthy.
How we built it
Django and PostgreSQL carry the generate-and-verify loop and store every candidate, kept or discarded. Flutter Web renders the corpus: the generated, kept, and discarded counts, the discard log with failure reasons, and the published exercises. A Docker sandbox judges each candidate in isolation, with no network, a read-only filesystem, dropped Linux capabilities, a non-root user, a process limit, a memory cap, and a hard timeout that kills infinite loops. I built the core loop with Codex and GPT-5.6 inside one Codex thread, moving from an empty schema to a loop that passes correct solutions, fails wrong ones, and kills hanging code.
Challenges we ran into
My hackathon credits turned out to be Codex credits rather than OpenAI Platform API credits, so a live generation endpoint was never funded. Rather than abandon the thesis, I moved generation into Codex itself and kept the sandbox as the only judge. The product did not change. Codex generated the candidates, the sandbox ruled on them, and the corpus carries the same proof it always would have.
Accomplishments that we're proud of
The sandbox catches real failures from a frontier model. One candidate asked for a function to validate an identity matrix, and the model's own solution shadowed a loop variable and crashed with a TypeError. The problem read fine and the tests read fine. Only execution caught it, and that candidate greys out in the discard log where a judge can see the whole thesis at once.
What we learned
The discard rate I expected did not survive contact with data. I assumed generation would fail often and verification would filter aggressively. The real rate was under two percent. That reframed the product from a filter into a guarantee: not "we throw most of them away" but "every exercise carries proof, and proof is not free." A measured finding beat the pitch I started with.
What's next for Whetstone
Broaden beyond Python, add difficulty calibration from student outcomes, and let instructors seed topics and review the discard log as a first-class teaching signal.
Log in or sign up for Devpost to join the conversation.