PROMPT GYM

Inspiration

I use AI every day, and I kept making the same mistake: a prompt would work on the one example in front of me, then fail on a real input with a missing order number, two requests in one message, or an amount hidden inside a delivery fee. Reading prompt-engineering guides didn't fix that, because reading about prompts isn’t the same as watching one break. I wanted a place where the failure is the lesson: one prompt, six cases you can’t see, and a score that moves when you change the prompt. The moment I most wanted to make visible was the overfitting trap — the fix that saves one case and breaks another. That “wait, my score went down?” moment is the product.

What it does

Prompt Gym is a no-sign-up practice game for prompt writers. Pick one of three packs — Structured Extraction, Following Rules, or Rewrite Without Losing Meaning — read the objective and criteria, write a prompt, and run it against six hidden cases. Results stream back one case at a time, with the deciding check, the response (when it’s safe to show) and the reason it passed or failed. Five of the six criteria are decided by deterministic rules; one uses a constrained judge that sees the criterion, the message and the response but never the user’s prompt. When an edit improves one case and breaks another, the app shows the observed flip, the delta and the changed lines — and only proposes a cause when the evidence supports it. Hidden inputs and the API key stay on the server. Attempts record the model, settings hash, evaluation version and fixture hash, and the app says “not directly comparable” when those differ. No accounts, no leaderboard, no stored prompts.

How we built it

I started with the Devpost Learn Skill Pack in Claude Code. 1-start interviewed me about the idea; 2-scope cut it to one loop; 3-prd and 4-spec wrote the behaviour and technical contract before implementation; 5-build worked in small slices, each ending in a runnable check; 6-ship prepared the submission. The scope, PRD and spec are in the public repo under devpost/. The app is Next.js 16, React 19, TypeScript, Tailwind and Zod, deployed on Vercel. The model under test is openai/gpt-oss-20b through Groq’s OpenAI-compatible API at temperature 0; the judge is openai/gpt-oss-120b. Fixtures are injected as a sensitive production environment variable, never committed and never sent to the browser. There is no JSON mode, schema or example added for the model under test — that would answer the Structured Extraction challenge on the user’s behalf. The constraint has to come from their prompt. I added 145 unit tests and three Playwright flows, and verified the live deployment with real runs, budget refusals and a real regression.

Challenges we ran into

The first provider key had no credits. I kept the same contract and moved to Groq, which also meant learning its strict structured-output constraints.
The judge is the weak point. A model judge can pass a vague answer, and its reason is not ground truth. I constrained it to one criterion, hid the user’s prompt from it, and made its output a typed hypothesis, not a fact. I did not claim judge accuracy.
Free-tier limits produced intermittent unresolved cases. I classified provider failures, added one bounded retry, capped concurrency and set a daily budget. Unresolved is a technical state excluded from the score, never a failed case.
Protecting hidden cases without removing the evidence. Redaction is proportional: a response is withheld only if it copies the whole input or an eight-word span. Rewrite cases opt out of span redaction because a rewrite is derived from its input by design; without that, a live run withheld all six rewrite responses.
Resisting the urge to explain every regression. A cause is shown only when the runs are comparable, exactly one case flipped and the edit is localised to at most four lines; otherwise the app reports the observation and says no cause is proposed.

Accomplishments that we're proud of

A complete loop that runs end to end with real model calls, not a static mock: landing, workspace, streamed results, attempt history, a lesson and a Markdown export.
A genuine regression observed live: 6/6 → 5/6, exactly one case flipped; the UI reports the observation and the −1 rather than inventing a story.
Five of six criteria are deterministic, and the one judge never sees the prompt.
Hidden inputs and the API key never reach the client; the production fixture set is a sensitive env var, and a clone without it falls back to clearly labelled sample cases.
Honest states: uncertain leaves the denominator, unresolved is a technical failure, and comparisons are gated on matching model/settings/evaluation/fixtures.
145 unit tests, three browser flows, typecheck and production build passing; the app is live and the repo is public under MIT.
The whole thing was planned with the Learn Skill Pack before code, and the planning docs are in the repo.

What we learned

Planning before code is not ceremony. The scope interview and PRD kept me from building a platform when the product was one loop.
Generating a score is easy; keeping the score honest is the product. Rules where a rule can decide, a judge only where it can’t, and a clear line between what was observed and what was inferred.
A regression is a fact; its cause is a hypothesis. Showing the flip without over-explaining it is more useful than a confident wrong story.
Hidden test data and learner evidence pull in opposite directions. Redaction has to be proportional or the learner has nothing to reason from.
Provider limits are part of the product. An unresolved call must be visible as a technical state, not silently scored as a failure.
The overfitting trap works when cases are diverse and requirements are consistent, not when the scoring is rigged. A genuinely general prompt can reach 6/6.
The Learn Skill Pack made the process legible: I can point to the scope, PRD, spec, build slices and ship checklist that produced the app.

What's next for PROMPT GYM

The biggest open question is whether it helps people improve. The mechanism is verified, but there is no user study yet; that is next, with a pre/post task and a control.
Calibrate the judge: repeated identical calls, agreement rates, and a stricter rubric before claiming accuracy.
Shared rate limiting and budget with Upstash Redis so limits are global, not per serverless instance.
A classroom pilot with an educator, and a fourth pack only if it earns its place.
More accessibility and real-device testing; the automated checks pass, but a human review is still needed.
Connect the public repo to Vercel for push-to-deploy, and keep the production fixtures private.