Loom

"Understanding, out loud."

Comment

Inspiration

I understand things better when I talk them through out loud than when I read a long passive explanation. Most AI tutors just hand you text and expect it to stick. I wanted something that works the way my mind actually works: I ask, it explains, and when a concept is better seen than described, it builds something I can look at and touch — live, while we're talking.

What it does

Loom is a voice-first learning environment. You speak to it, it speaks back, and during the conversation it decides whether the idea needs a visual: a manipulable simulator for quantitative relationships, a connection map for discursive or relational ideas, or a plot board for simple formulas and data. When no artifact would help yet, Loom stays on the board and simply notes a concept, a link, or a source. The principle behind it: the medium adapts to the content, not the other way around.

How we built it

I started by writing a structured brief before any code was written, and required Codex to propose a phased build roadmap and wait for my confirmation rather than start immediately. I set the build order deliberately: the projectile "mother scene" first, because it's the heart of the demo, the riskiest piece technically, and it gives reusable UI scaffolding for everything after it.

We built mock-first: a fully scripted, deterministic version of the two demo sessions, so the experience could be validated and the video had a safe fallback before any real API call existed. Only once that was stable did we wire in the real OpenAI stack — STT and TTS for the voice layer, and two GPT-5.6 tiers doing different jobs: a fast tier (Terra) handling the real-time spoken dialogue, and a stronger tier (Sol) deciding, turn by turn, which comprehension artifact — if any — fits the concept, and producing a validated structured plan for it.

Every artifact is controlled rather than freely generated: the model never outputs raw coordinates, SVG paths, or executable code. It returns bounded, validated data (an angle and a velocity, a set of labeled nodes and links, a formula and its axis ranges), and a local renderer owns all the drawing. That boundary was a deliberate safety choice, made early, that shaped almost every decision after it.

Once the two-artifact scene was solid, I pushed to go beyond a fixed script: I asked Codex to let the model reason about topics beyond the original examples and choose among the controlled renderers, rather than being told in advance which one to use.

Challenges we ran into

  • Making structured model output reliable enough for a live demo: schema validation, one retry, and a stable mock fallback when an API response is malformed, unavailable, or too slow.
  • Balancing genuine flexibility against demo safety: letting the model reason freely about what to show while keeping how it's drawn fully controlled, so nothing could break live on camera.
  • Getting the model's spoken reasoning ("why" it's choosing a given form) to happen before the artifact appears, not as an afterthought caption — that ordering was what made the "medium adapts to content" idea actually visible instead of just asserted.

Accomplishments that we're proud of

  • A voice-first learning tool where the model selects the learning form rather than following subject-hardcoded routes, while every visual artifact remains controlled and reliable.
  • Verifying that generalization myself, live, on a topic never scripted into the build (French Revolution), and watching Loom correctly choose to show nothing rather than force an artifact.
  • Keeping the whole system honest about its own limits: Loom doesn't pretend to be a fully arbitrary canvas, and says so plainly in its own documentation.

What's next for Loom

Today Loom reasons generally but draws through a small set of controlled renderers — a projectile simulator, a connection map, and a plot board. The natural next step is to widen that library, and further out, to explore a more open, sandboxed way for Loom to draw any concept live — closer to a real teacher's blackboard — with the validation and safety guarantees that would take to run unsupervised. I'd also like to bring back ideas we deliberately scoped out for the hackathon: a downloadable session summary, and — longer term — the ability to notice connections across separate study sessions over time.

Built With

Share this project:

Updates