Track: Track 3 — Agent Society
Inspiration
Students pay for course subscription after subscription and finish almost none of them — then fall back on their own dense university slides and PDFs that are impossible to understand alone. The AI tools meant to help just ask one model to imagine a lesson: hallucinated facts, made-up animations, passive watching. We wanted the opposite: a teacher you can check — where every animation comes from code that really ran, and every claim points at your own source.
What it does
Bring any material — a PDF with figures, a web article, pasted notes, a photo of a slide — and a society of Qwen-powered AI teachers turns it into a narrated, interactive, Udemy-style course:
- A Dean plans the episodes and lessons; each lesson is built by its own crew of agents, streamed live to the Studio over SSE so you watch the faculty debate as it works.
- The tutor writes on a board in sync with its voice — one audio clock drives the board, the code panel, karaoke word-highlight subtitles, and quizzes (a quiz pauses the clock until you answer; three kinds — multiple choice, graded scenario, and Feynman-style teach-back — and every lesson is required to carry one).
- The signature feature: algorithms are animated from really-executed code. The engine runs the algorithm in a network-isolated sandbox, records structured step events, and drives the screen from that recording — active line, pointers riding the array, visited sets, a step-by-step trace table. A dry-run scene that cannot produce a real execution trace refuses to ship — no fabricated animation, ever.
- "Try your own input": on a dry run, the student edits the call (their own array, graph, or arguments — even toggling memoization on a recursion) and the whole animation re-derives from a fresh deterministic run of THEIR input, across 9 trace engines, with no LLM in the loop.
- Ask the Tutor, mid-lesson: playback holds while you type; a Qwen agent answers grounded in the current scene and ends with a Socratic follow-up. A practice panel on the final scene checks answers mechanically and a diagnosis agent explains exactly what went wrong, with graduated hints.
- Progressive playback: the first scenes start playing while the society is still writing the rest ("building live · N/M scenes ready", with honest stall recovery).
- Real figures are lifted from your PDF (MinerU: figures, tables, LaTeX, page renders) and shown with a "Source · page N" stamp; annotation marks are placed by Qwen-VL reading the actual pixels (any mark it can't locate is dropped, never guessed); an independent auditor blocks anything your source doesn't support — and every scene carries "The Society's Work" receipt: the actual debate transcript (proposals, objections, repairs) with per-claim citations.
- The student does, not watches: edit and re-run the lesson's code against the server sandbox, or entirely in the browser (real CPython on WebAssembly via Pyodide in a Web Worker).
- Beyond algorithms, the Board Director can call for interactive scene types: PhET/Desmos simulations, rotatable 3D molecules, an in-browser SQL playground (real SQLite), live data charts, a steerable gradient-descent explorer, a terminal view, and drag-to-predict manipulables (predict first, then drag the slider and watch the curve recompute from the engine's formula).
- Learning sticks: SuperMemo-2 spaced repetition with a forgetting-curve view, streaks + a 12-month activity heatmap + achievement badges, bookmarks that capture the exact second and teaching context (with seek-bar markers and keyboard shortcuts), and notebooks that write back — grounded synthesis, a knowledge graph, handwritten ink boards, teach-back and improve actions.
It's open source (AGPL-3.0) and runs on your own Qwen usage — no subscription.
A note on the demo video: 3 minutes only fits the core loop — upload → society debate → the executed-code animation. Every feature described on this page is implemented, wired in the app, and testable in the repo and the live deployment.
How we built it — the agent society (Track 3)
Task division. Each agent is one job in one file under lib/orchestration/agents/: a Domain Router classifies the material and picks exactly ONE planner — the Coding Instructor, one of 15 specialist domain Teachers (math, physics, chemistry, biology, ML/AI, agents/RAG, databases, networking, OS, architecture, SRS, SQA, history, law, economics), or the Universal Teacher. For a full course, the Dean plans episodes and fans out one queue job per lesson (BullMQ on Redis), so lessons generate in parallel across workers.
Dialogue & negotiation. Every scene's board goes through a real LangGraph StateGraph review cycle: the Board Director proposes; a Grounding Auditor (hard gate) and a Pedagogy Critic (advisory) audit in parallel; the Director revises against their objections. The blackboard is an append-only log of six typed message kinds — proposal, objection, evidence, revision, verdict, handoff — and an objection with no evidence is rejected by code ("no evidence, no objection").
Conflict resolution. Debate is bounded by MAX_DEBATE_ROUNDS (default 3). If grounding objections survive at the cap, the Arbiter issues a binding verdict — the message schema itself enforces that only the Arbiter may issue verdicts — with a strict-consensus fallback (every objected object removed) if the Arbiter's own call fails. Only the failed stage re-runs.
Real execution. For coding scenes, the Execution Tracer + Code Runner write and run the real program in the sandbox (Judge0, or Docker with --network none --memory 256m --cpus 1 --pids-limit 128). The traced run emits step events; 15 behavioral detectors then pick the teaching lens from the recording itself — DP grid, graph walk with per-node state (Tarjan's disc/low riding under the nodes), heap, trie, union-find, recursion tree, linked list, intervals, and more — with zero per-problem code. Authority on any disagreement: execution > structure > behavioral invariants > AI interpretation.
One model per job — all agent intelligence on Qwen Cloud (DashScope):
| Agents | Model |
|---|---|
| Dean · 15 Teachers · Coding Instructor · Arbiter | qwen3.7-max |
| Board Director · Voice Writer · page/slide vision | qwen3.7-plus |
| Domain Router · Grounding Auditor · Pedagogy Critic | qwen3.6-flash |
| Execution Tracer · Code Runner | qwen3-coder-plus |
Every call goes through one client (lib/qwen/client.js) with a per-agent token ledger.
The measurable gain over a single agent (Track 3 requirement — measured, not asserted)
- Mechanical validators, 4 matched coding problems (
eval/society-vs-single.eval.mjs, results JSON in repo): the single-agent baseline's hand-written dry runs fail the elite quality gate on all 4 (e.g. "0 steps carry pointers") and ship 1 structural contract violation; the society's traces show 0 contract failures across all 4. - Blind pedagogy rubric (
eval/RESULTS.md): 7 criteria judged in BOTH presentation orders — a win only counts if it survives the order swap. The society wins 4 and 5 of 7; the single agent wins 0. - Universal dry-run engine: 63/64 (98%) structural-elite, 0 errors, zero per-problem code on a 64-problem LeetCode battery spanning arrays, trees, graphs, heaps, tries, DP, and design (2026-07-15 run; the battery has since grown to 68 problems). The one floor case is Euclid's GCD — pure arithmetic with no structure to draw.
- 793 passing tests (
npm test, no tokens spent). - Honest tradeoff, reported plainly: the society spends far more tokens and wall-time than a single agent (e.g. 224k vs 7.4k tokens on one benchmark topic) — the price of validation, real execution, and grounded debate.
Challenges we ran into
- Making "no hallucinated animation" structural, not aspirational — the fix was a hard gate: a dry-run scene without a real ExecutionTrace throws instead of shipping, and hand-authored traces are stripped by the validator.
- One lens for thousands of problems — instead of per-problem visualizers, we built record-once/detect-later: one real execution, then behavioral detectors choose the view. Adversarial rounds on unseen algorithms (8-node Tarjan, Prim's MST) exposed mis-lens bugs that are now impossible by construction.
- Keeping debates honest — evidence-required objections and an arbiter that only exists for deadlocks; otherwise critics either rubber-stamp or filibuster.
- Production reality — slow regional model pools, per-scene concurrency, worker fan-out with idempotent job ids and a priority lane for the first playable lesson, a second-chance rescue pass that re-runs scenes lost to transient provider errors, honest failure recording for dropped scenes, and timeline reconciliation so voice and board can't drift.
What we learned
A single model asked to "make a lesson" will confidently fabricate the parts students trust most — the animations and the numbers. The gains came from structure: one agent per job, critics that must attach evidence, a gate that prefers honest failure over plausible output, and an execution engine so the screen shows what the code did, not what a model imagined.
What's next
Human-in-the-loop control on every scene — the teacher (or student) approves, edits, or sends any scene back to the society for regeneration before it ships, turning the debate receipt into an interactive checkpoint. Also: swapping the media seam to Alibaba OSS (the code already writes through one seam), a persistent accumulating board, and richer non-coding visual engines.
Beyond the player — the full study system shipped around it
Notebooks — a second brain that writes back. Press N during any lesson and the exact moment (timestamp + the line being taught) is captured into your notebook; "Lesson pages" arrive half-written from what you saved. Ask the notebook to synthesize and a second LangGraph agent chain runs live over SSE: retrieve (vector RAG — Qwen text-embedding-v4, cosine over your own blocks) → planner → evidence gate → parallel self-citing writers → citation gate → illustrator → a reviewer that drops any sentence unfaithful to your sources. Every block has actions: improve, narrate (voice), Feynman teach-back, attach evidence, and run a dry-run on code blocks. Plus a knowledge graph across notebooks and handwritten "visual notes" on a real ink engine (lasso/selection geometry mirrored from Xournal++).
Focus Guard — Qwen watches your focus, not just your lessons. A Chrome (MV3) extension + companion server: as you study, each page you visit is classified — a deterministic offline fast-path first, then Qwen vision (qwen3.7-plus) reads the screenshot + page text against your stated goal — into study / partial-study / non-study, and writes a specific, goal-aware nudge (overlay chat + refocus timer) to pull you back. The dashboard scores the whole session page by page, with per-page confidence; sessions persist per device, owner-scoped.
Audio → Notes — lectures become study notes while you sit in class. Three modes: live listening (browser speech recognition), upload a recording (transcribed by faster-whisper on the companion server), or paste a transcript — then Qwen structures the raw transcript into clean, organized study notes.
Progress — models what you'll forget, not just what you did. Every lesson's quiz becomes a real SuperMemo-2 deck at save time; bookmarked moments get their own spaced-review track (grade a memory and its interval grows). The dashboard shows the forgetting curve with the review scheduled at the dip, memories due now, streak + best streak, a 12-month activity heatmap, and achievement badges computed live from real activity — no vanity numbers. Bookmarks keep the exact second and the taught line, marked on the seek bar, one click to replay.
A note
Voice note: the codebase ships a Qwen3-TTS adapter (qwen3-tts-flash) as the code default; the demo deployment opts into ElevenLabs for word-level timestamps that drive the karaoke subtitles. All reasoning agents are Qwen.
Log in or sign up for Devpost to join the conversation.