Chalkbox

A teacher types a misconception; Codex builds, self-tests, and publishes a live interactive manipulative students open on their phones.

Comment

Inspiration

It's 9 PM on a Sunday. Ms. Alvarez teaches 6th-grade math to 32 kids at three ability levels, and she knows exactly which misconception will sink half her class tomorrow — that dividing by a fraction makes the answer smaller. She can write a worksheet. She can find a video. But the one thing that would actually break the misconception — something her students can touch and drag until it clicks — is software, and she cannot write software. The research is unambiguous that manipulatives beat explanations for misconceptions; they're also the one artifact teachers can't author. Chalkbox exists to close exactly that gap, and nothing else.

What it does

A teacher types the misconception she wants to break — "show why dividing by a fraction makes the answer bigger, not smaller" — and in about a minute she has a live, draggable manipulative her students open on any phone. No code, no app store, no vendor roadmap.

What makes it real is the loop no chatbot can run. At product runtime, Codex doesn't describe a manipulative — it writes a single-file interactive React component plus the smoke test that proves it teaches the right thing, runs it headlessly, asserts the interactive invariants hold (drag the divisor smaller → the quotient must grow), retries with the error trace on failure, and only then publishes. Nothing untested reaches a child.

Three surfaces, deliberately nothing more:

  • Create — the prompt box streams an honest build timeline with real timestamps: safety gate → Codex writes → static checks → self-test → Verified · Published.
  • Gallery — 15 seeded manipulatives, each showing its real Common Core / NGSS code and the exact teacher sentence that made it.
  • Student link — phone-first, zero-chrome, no login; generated code ships under a strict no-network CSP.

Scope is proudly narrow: math and physics manipulatives only. That discipline is why the verification loop can be hardened enough to trust with 32 kids.

How we built it

The generation engine was built in a Codex CLI session (the submitted /feedback Session ID) — fitting, because the product is the coding-agent workflow, productized for someone who can't code:

  • Four verification gates. G1: GPT-5.6 Luna triages every prompt (classroom safety, grade band, CCSS/NGSS alignment) before any generation credit is spent. G2: AST-level static validation (import allowlist, no network APIs). G3: the component is mounted headlessly and its Codex-authored probe drives it — interactive invariants must hold. G4: an output-safety scan on the rendered text, then publish.
  • GPT-5.6 in two tiers via the raw Responses API: Sol (high reasoning) writes and repairs the component+probe pair; Luna (fast/cheap) runs the gate. The server-only SSE route keeps the key off the client.
  • The probe cross-check is the anti-fake moat: requireRenderedTestIds verifies the probe's controls actually exist in the rendered markup, so Sol can't "pass" a test it invented for a different component.
  • Next.js 15 / React 19 / TypeScript, Tailwind v4, deployed on Vercel; 28 unit + 26 Playwright E2E tests, 6-stage CI with CodeQL.

Challenges we ran into

  • The engine that "worked" but never ran. Our first "real" engine silently fell back to the seeded stub on every request — CI green, product a lie. A bench honesty-guard (detect the stub's fixed simId, exclude it from the success rate) exposed it before fake numbers shipped. Root causes: the raw Responses API nests output text differently than the SDK, per-attempt timeouts too tight for high-reasoning generation, and Sol inventing its own invariant schema until the prompt pinned the exact one.
  • Verifying generated pedagogy, not just rendering. A sim that renders but teaches the wrong thing is worse than no sim. The fix: Codex must emit the component and its invariant probe together, and the probe is executed against the real DOM — with the cross-check above so the pair can't drift apart.
  • A CSS compiler fallback that flooded judged surfaces. Our design tokens used color-mix(...alpha, transparent); the compiler's compatibility fallback strips the alpha, turning subtle brand glows into opaque paint in real browsers. We purged color-mix from the codebase for literal rgba and byte-diff-tested every hover state with Puppeteer.
  • Honest demo mode. The site must be judge-testable with zero keys, without ever pretending. Demo mode is labeled on-screen, live mode is an explicit double opt-in (key + flag), and spend is budget-guarded.

Accomplishments that we're proud of

  • The self-test loop is real and on camera. The demo video's generation is a genuine live run — prompt to Verified · Published in ~53s, and the fail → read-the-trace → fix → verified path is computed, not staged.
  • Zero-setup judging. The live site works with no login and no keys; every gallery card carries a real curriculum standard and its generating sentence.
  • An honesty pipeline, not just a demo: published success rates come from a reproducible bench that refuses to count stub fallbacks.
  • 54 automated tests, CodeQL, and a CI gate that forbids live keys in test env.

What we learned

  • An agent that can execute and test its own output is categorically different from one that can only write it — the retry-with-trace loop is where the value lives, and it's exactly what a non-coder can't do themselves.
  • Verification needs adversarial design: generated tests must be structurally bound to the generated artifact, or the generator will satisfy itself.
  • Honest constraints (narrow scope, labeled demo mode, published failure rates) are a feature judges — and teachers — can feel.

What's next for Chalkbox

  • Interactive client rendering of newly generated sims (server-verified today, with the sandboxed-iframe host in progress) — then remixing: fork any gallery sim with a new sentence.
  • Teacher pilots: put Chalkbox in front of 5–10 real teachers and publish the bench success rate (p50/p95 latency) on the site.
  • Broader invariant DSL (energy conservation, monotonicity families) while keeping the math+physics-only discipline.

Built With

  • codex
  • github-action
  • gpt-5.6
  • next.js
  • node.js
  • openai
  • playwright
  • react
  • responses-api
  • supabase
  • tailwindcss
  • typescript
  • vercel
  • vitest
Share this project:

Updates