rubric stress tester

Make your rubric agree before your graders have to. It finds the criterion two graders would score differently, proves the split with real answers, and rewrites it to remove the ambiguity.

Comment

Inspiration

Anyone who has ever graded a stack of papers with someone else knows the argument. You gave it a 4, they gave it a 2, and the frustrating part is you're both right. The rubric said "uses strong evidence" or "clear and concise," and it turns out the two of you were quietly grading against slightly different ideas of what those words meant. Nobody was being unfair. The rubric just never actually decided.

And you never catch it in the moment. You catch it weeks later, when it comes back as a regrade request, or a parent email, or a student who feels like their grade came down to which TA they got. We kept thinking: that argument shouldn't happen after the grading. It should happen before, so the rubric settles it instead of the people.

What it does

It's meant to be dead simple. You paste in a single line from your rubric, one criterion, and hit go. From there it:

  1. Points at the problem word. It finds the one phrase two reasonable graders could read differently, and spells out both readings in plain terms. Not "a harsh grader vs. an easy grader," but two honest interpretations of what the words actually ask for.
  2. Proves it's real. It writes two believable student answers and grades each one under both readings, so you watch the scores split. That's the moment it stops being a hunch and becomes a number.
  3. Fixes the wording. It rewrites the criterion into something concrete you can actually check, broken into small 1-point pieces so no single fuzzy word can swing much.
  4. Checks its own work. It re-grades the same two answers against the new wording and tells you how much room to disagree is left.

Paste that rewrite back in and you watch the gap close to nothing: "No reliable divergence: graders will agree." And it talks in points, not percentages, because "3 of 5 points could change depending on who's grading" is a sentence a teacher feels immediately.

How we built it

Codex built the whole thing start to finish, with GPT-5.6 doing the thinking. Under the hood it's a Next.js 14 / TypeScript app, and every step is a separate GPT-5.6 call with a strict JSON schema so the output is always structured: spot the ambiguous phrase, write the two answers, grade and critique them, rewrite the criterion, then re-check. All the model calls happen on the server, so the API key never touches the browser. It's live on Google Cloud Run, with the key kept in GCP Secret Manager and handed to the app only at runtime.

Challenges we ran into

Finding ambiguity turned out to be the easy part. The thing that nearly broke us was getting the rewrite-and-recheck loop to actually get better each time. In the early versions it would sometimes go the wrong way: you'd ighten a criterion and the disagreement would come back bigger, because the rewrite had lumped most of the points into one chunky piece that then hid a new, larger ambiguity. Watching a fix make things worse was demoralizing.

What finally cracked it was one simple check the code makes on its own: has this criterion already been broken into parts? If it hasn't, the tool goes in aggressive and hunts for the vague phrase. If it has, it gets cautious and refuses to split hairs. And every rewrite is forced into 1-point pieces. Once no piece can carry more than a point, the worst any disagreement can ever move is one point, which means tightening can't backfire and always walks down to zero.

Accomplishments that we're proud of

  • It actually converges. For a long stretch, it didn't. We'd tighten a criterion and the disagreement would sometimes get worse instead of better, which was maddening. Getting it to shrink to zero every time, and never bounce back up, is the thing we sweated the most and the thing we're happiest about.
  • It shows its work. It doesn't just tell you a criterion is ambiguous and ask you to trust it. It writes two student answers, grades them under both readings, and lets you watch the two scores come apart. You don't have to take our word for it.
  • It counts in points, not vibes. We report how many points could swing depending on interpretation, which is exactly what a teacher is already worried about. And when a criterion is genuinely fine, it says so, instead of inventing a problem to look clever.
  • You can actually use it. It's not a notebook or a slide deck. It's a real web app, live right now, that any teacher can paste a criterion into and get an answer.

What we learned

The stuff that made this good was mostly the stuff we said no to. No "strict grader vs. nice grader" personas, just two honest readings of the words. No abstract ambiguity score, just points. No hanging onto a shaky example, the critic throws it out. Every time we picked the more disciplined, less clever option, the tool got more trustworthy. That was the real lesson: for something teachers are supposed to lean on, boring and precise beats smart and flashy.

What's next for Rubric Stress-Tester

We'd love to let a teacher drop in a whole rubric at once instead of one criterion at a time, pull rubrics straight in from Google Classroom or an LMS, and hand back a clean before/after report they can attach when they revise a rubric, so there's a record of why the wording changed.

Built With

Share this project:

Updates