LessonWorld

Inspiration

AI can draft a lesson that looks complete—clean copy, a quiz, maybe a diagram—and still fail the moment a real learner opens it. Controls don’t respond. The checkpoint never transitions. Labels clip on mobile. Citations don’t resolve. A polished preview hides all of that.

We were inspired by a simple question: what if lesson generation treated the browser as part of the loop, not an afterthought?

That became LessonWorld: Predict before you publish. Instead of shipping the first fluent draft, we generate several lesson designs, predict which one will survive a real browser pass, spend one ephemeral sandbox on the strongest candidate, and only mark a lesson Verified when execution evidence agrees.

What it does

LessonWorld turns a topic and learning objective into interactive STEM lessons with an evidence gate:

1. Compose — topic, objective, difficulty, optional source (or a prepared fixture).
2. Mental rollouts — four validated LessonSpec candidates; a dynamics model predicts publishability, score, blockers, and confidence for each.
3. Human approval — CopilotKit / AG-UI locks the selection and asks for an explicit “run in Daytona” approval.
4. Reality check — Daytona spins up a fresh sandbox; a Playwright Learner QA harness checks render, layout, controls, citations, and checkpoint behavior.
5. Compare & verify — predicted vs. actual outcome, Braintrust traces, optional bounded repair, and a Verified badge only when the gate passes.

Behind the demo is a training loop: Daytona trajectories feed smaller Fireworks models—a student policy that authors structured lessons, and a lesson dynamics model that predicts sandbox outcomes so we can rank candidates cheaply before paying for a real run.

How we built it

We separated prediction from truth, and UI from authority:

• Schemas first — every candidate is a validated LessonSpec (structured JSON), rendered by server-owned templates—not model-authored executable code in the browser.
• Daytona as the reality oracle — pinned snapshots, ephemeral sandboxes, fixed harness commands, canonicalized LessonOutcomes. User input is lesson data, not sandbox authority.
• Fireworks for policy + dynamics — SFT views built only from Daytona-verified, training-eligible transitions (synthetic candidates stay labeled unverified).
• CopilotKit / AG-UI — typed agent events for candidates, predictions, approval interrupts, Daytona lifecycle, and predicted-vs-actual comparison.
• Braintrust — traces, scores, and experiment comparisons so judges (and we) can see why something passed or failed.
• Offline-safe demo path — fixtures and labeled replay so the story works without live credentials; live providers stay behind explicit approval gates.

Stack highlights: FastAPI backend, React/Vite frontend, Daytona + Playwright verification, Fireworks training/inference adapters, CopilotKit AG-UI, Braintrust evals—plus a fuller learner app path (course gen, tutor, A2UI tools) isolated from the training pipeline.

What we learned

• Looking good ≠ working. Static LLM output is a weak signal for interactive learning UX.
• World models earn their keep when actions are expensive. Predicting four outcomes and verifying one beats verifying everything.
• Provenance is a product feature. Training eligibility, canonical hashes, and “Verified only after Daytona” keep the demo honest.
• Human-in-the-loop should be narrow. One allowlisted approval (“run this candidate”) beats an open-ended agent with shell access.
• Sponsor tools compose best when each owns one job: Daytona = execution truth, Fireworks = cheap prediction/authoring, CopilotKit = control plane, Braintrust = eval evidence.

Challenges we ran into

Partial runs vs. real truth. Without Playwright in the sandbox image, the harness correctly returns partial—and those rows must not become training data. Teaching ourselves not to “fix” empty training views by relaxing the gate was a constant discipline check.

Determinism under cloud nondeterminism. Sandbox IDs, timestamps, and preview URLs want to leak into hashes. We had to canonicalize aggressively so datasets and comparisons stay reproducible.

Safety vs. spectacle. It’s tempting to let the model emit HTML/JS and “just run it.” We kept a hard boundary: structured specs in, server-owned renderers out, no arbitrary user commands in Daytona.

Prediction–reality mismatch. The dynamics model can be wrong; the product has to celebrate that comparison, not hide it. Building UI that locks selection before revealing the actual outcome was harder than another chat transcript.

Cost and approval gating. Live Daytona + Fireworks jobs burn money. Everything billable sits behind mode flags, plan hashes, and explicit approvals so offline demos don’t accidentally become production spend.

Scope. A full learner product and a training/verification pipeline both wanted attention. We kept them isolated so hackathon claims about Daytona → Fireworks trajectories stay clean and auditable.

What’s next

Scale verified STEM transitions beyond the hackathon package, tighten dynamics-model ranking, add richer pedagogical evaluators on top of runtime checks, and close the loop so failed reality checks feed better policy repairs—still with the same rule: no Verified badge without environment evidence.

Built With

• braintrust
• copilotkit
• daytona
• fireworks
• python
• react