Model Jury

An evidence-first game for students where they learn content and use it to take a flawed AI to court. The students prosecute the AI by exposing unsupported claims, testing repairs, and eventually win.

Comment

Inspiration

Most AI-literacy exercises use obvious hallucinations or generic warnings. Students learn to distrust AI, but not how to evaluate a mostly correct answer. I wanted to change that by creating a new way for students to learn subjects while also improving their AI literacy in a fun, game-like way!

What it does

First of all, it is important to realize that I only built this as a demo lesson. Usually, teachers would choose from various subjects for students to learn. For our demo-lesson, students use real photographs to learn diagnostic traits, heritable variation, and natural selection before comparing their reasoning with BioGuide.

BioGuide is realistic on purpose: four of its five field notes are scientifically accurate. However, one answer extends a true idea slightly beyond what the evidence supports. Students must identify that exact sentence, connect it to the photograph and source note that expose the problem, and write a stronger explanation.

That evidence becomes a reusable practice case and a versioned repair. Students compare before-and-after results, reveal more hidden flaws of Bioguiide, complete a biology field report, and optionally enter a dramatic four-phase court-like battle. The courtroom has evidence rules, credibility, boss phases, original Web Audio cues, and assignment-scoped remedies. Correct answers remain in the record. So, removal requires a published burden of proof.

Teachers get a separate four-step studio for defining a classroom task, learning goals, source packet, controlled practice failures, and a mandatory review gate for an AI to create lessons for them.

How I built it

The application uses Next.js 16, React 19, strict TypeScript, vinext/Vite on a Cloudflare-compatible runtime, Zod, Zustand, Motion, CSS Modules, Vitest, Testing Library, and Playwright. A deterministic no-key mode makes the complete judging path repeatable. A server-only OpenAI provider boundary uses the official JavaScript SDK and Responses API, Structured Outputs with Zod validation, bounded retries and timeouts, trace IDs, redacted errors, and store disabled.

The local evaluation engine scores evidence grounding, uncertainty calibration, misconception avoidance, and helpfulness. It runs visible and hidden assertions against immutable agent versions and detects when a repair becomes uselessly cautious. The trial engine applies assignment-specific removal standards without pretending to certify an entire model.

How Codex and GPT-5.6 helped

Model Jury was built during OpenAI Build Week with Codex powered by GPT-5.6 as the primary engineering partner. GPT-5.6 helped reason across the connected educational flow so the biology lesson, subtle controlled failure, evidence notebook, repair tests, field report, and courtroom remedy stayed consistent as the product evolved.

Codex mapped the domain and state model, implemented the student and teacher journeys, built the deterministic evaluation and trial engines, iterated on the visual system, and ran repeated student and teacher playtest loops. It also drove browser QA across desktop and mobile, accessibility checks, regression testing, and the public release workflow.

To keep judging fair, the public app uses a fully deterministic simulation by default and clearly labels any simulated model content. The live OpenAI integration is built and tested, but this version does not claim that it was run with real production credentials.

Challenges

The hardest challenge was making the assistant wrong in a way that felt authentic. The issue could not be a cartoon hallucination. It had to extrapolate from a true scientific idea while remaining small enough that students needed the lesson to catch it. We also had to ensure the optional courtroom made evidence reasoning more fun without turning the product into a generic anti-AI game.

A second challenge was keeping every artifact connected. The photograph, student observation, exact weak claim, source note, practice case, revision, hidden transfer result, field report, and verdict all preserve their provenance.

Accomplishments

  • A complete, no-key student journey from lesson through field report.
  • A credible four-of-five BioGuide baseline with one subtle unsupported inference.
  • Traceable claim-to-source evidence and student-authored practice cases.
  • Versioned repairs, hidden transfer checks, and regression detection.
  • A visually distinctive four-phase courtroom capstone.
  • A review-first teacher compiler.
  • Responsive, keyboard-accessible UI with reduced-motion support.
  • 102 unit/component tests and 34 desktop/mobile end-to-end tests.

What I learned

Small AI mistakes are harder and more educational than obvious failures. Students need domain knowledge before they can evaluate an answer well. We also learned that a repair needs an anti-overcorrection test: adding uncertainty is not a win if the assistant becomes too cautious to help.

What's next

Next I would add teacher-reviewed cases for more subjects, richer classroom analytics, collaborative evidence review, and carefully evaluated live GPT-5.6 experiences while preserving the deterministic judge path and explicit human review gates.

Try it

Built With

  • codex
Share this project:

Updates