-
Every score cites the exact substring that drove it. 3 of 11 variants cleared threshold, 0 gate disagreements between judge and Python.
Inspiration
Every AI content tool generates. None of them verify. A creator gets ten titles back, has no idea which one is good, rewrites one by hand, and the tool saved them nothing. The unsolved half isn't generation it's knowing which output is worth shipping.
What it does
ShipScore takes a content brief and a target keyword, generates title and description variants, then scores every one of them blind against a frozen rubric. Each variant comes back with a composite score, a ship/gated verdict, and the exact substring that drove each dimension score. You get a ranked, auditable shortlist instead of raw output you have to re-review.
Batch mode runs a CSV of briefs in one command.
How I built it
Three invariants shaped the architecture:
The judge is blind. Separate API call, separate context. It never sees the writer's prompt or reasoning, and candidates are shuffled before every scoring call. This wasn't paranoia the eval later showed the first prompt slot scores about 4 points lower on average, so unshuffled scoring would have baked in a position bias.
No LLM ever computes a score. The model emits integers and evidence spans. Python does the weighting, the gates, and the threshold. That's what makes the output auditable.
Stability is measured, not assumed. Sonnet 5 removed sampling parameters, so "judge at temperature 0" isn't available. Rather than drop to an older model to keep the spec literal, I reframed it: frozen prompt, frozen rubric, schema-enforced JSON, and then I measured how consistent the judge actually is across two independent scoring passes.
Results
Judge self-agreement on the ship decision: 0.900 raw, kappa 0.756, over 160 candidates across two independent shuffles. Per-dimension exact agreement runs from 0.54 (hook) to 0.83 (overpromise).
Against human labels: n=60, two blocks of 30, single annotator, blind to all judge output. Raw agreement [RAW], kappa [KAPPA].
The two annotation blocks shipped at 67% and 73%, so the bar held across sessions. That matters: it means human-judge disagreement is genuine disagreement, not annotator drift.
Challenges
The honest one is that hook carries 30 of 100 points and is the least stable dimension at 0.54 self-agreement. That pairing drives most of the flips near the ship threshold. I froze the rubric at v1.0 before running the eval and did not retune it afterward, because retuning against your own eval invalidates the eval.
The 70% pooled ship rate also makes the human classes imbalanced, which raises expected chance agreement and pushes kappa down relative to raw agreement. Both numbers are reported for that reason.
What I learned
Building a verification layer forces you to make claims you can be wrong about. That's uncomfortable and it's the point. A generator can hide behind "looks good to me"; a judge with a published agreement number cannot.
I also learned that the design decision I was proudest of per-call shuffling — only proved itself because I measured for the bias it prevents. Without the eval it would have been a guess that happened to be right.
Limits
Single annotator at n=60 is a sanity check on the judge, not a validation of it. The rubric is opinionated and tuned for one niche. There's no live CTR data, so "ship-worthy" means my judgment, not measured performance. Two content types only.
Built during the hackathon
All code in the repo was written during the event window. Architecture and rubric design were planned beforehand, as the rules allow.
90 tests, all passing. Fresh-clone verified.
Log in or sign up for Devpost to join the conversation.