Vizli - Voice First Canvas

Inspiration

People who explain things out loud (teachers, tutors, presenters, anyone at a whiteboard) spend their attention on clicking, dragging and switching slides instead of on the explanation. We wanted the visuals to follow the voice: say "highlight the second row of A" and the right row lights up, say "change revenue to 4.5 million" and the number changes, say "explain how DNS resolution works" and a lesson draws itself while you keep talking.

What it does

Vizli is a presentation studio with editable slides and expanding whiteboards. Every scene is validated JSON with stable object and part IDs, so a spoken sentence can land on exactly the right bar, matrix cell or diagram node.

• Point, highlight, animate: "Point to revenue", "Highlight the second row of matrix A in blue", "Animate matrix A with draw effect". Temporary emphasis appears while you are still mid-sentence (measured 0.3–0.5 s after the words) and is confirmed by Nemotron when the sentence ends.
• Edit by voice: "Make matrix A 3 by 2", "Change Revenue to $4.5M", "Set row 3 column 2 of matrix A to 9". Edits are atomic, validated and one undo step.
• Narrated lessons: "A is three by two and B is two by three…" draws a matrix-multiplication lesson step by step; "Explain how DNS resolution works" asks Nemotron 3 Ultra to author a lesson that the server compiles and validates before it plays. The microphone keeps listening while a lesson animates.
• Whiteboards that grow: "Draw a software architecture", "Let's go right", "Go to Start", "Move Web client right". Named areas and a persistent camera keep earlier content in place.
• Sourced images: "An apple on the x axis" places a Wikimedia Commons image with attribution on the chart axis.
• Real-time collaboration: several people open one link on Macs and iPads; edits, pen annotations and pointers sync live, and each person keeps their own camera and undo.
• Honest by default: without credentials the app runs in a clearly labelled demo mode with no model inference. Ambiguous or hedged speech skips instead of guessing, and every action is fenced by the scene revision it was made for.

How we built it

• Three Nemotron tiers on Nebius Token Factory, called at runtime through the chat-completions API with response_format: json_schema (strict: true) and thinking switched off for interactive calls:
  • Fast: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B routes every final cue before any rule runs. It classifies the intent, repairs speech-recognition errors and declines tentative speech.
  • Core: nvidia/nemotron-3-super-120b-a12b resolves free-form visual actions and plans scenes and illustrations.
  • Reasoning: nvidia/Nemotron-3-Ultra-550b-a55b authors lessons for topics without a hand-written planner.
• NVIDIA-hosted Nemotron ASR (Riva streaming gRPC over TLS) turns microphone audio into partial and final transcripts.
• Deterministic tools that Nemotron selects: target IDs, matrix arithmetic, edits, camera moves and image search are trusted planners; the model never bypasses scene validation.
• Provenance on every action: each action records which model or rule produced it, with tier and latency, and the activity log prints it ("Nemotron 3 Nano routed “attention” 1.4 s → Nemotron 3 Super on Nebius 2.0 s").
• Stack: React, TypeScript, Vite, an SVG/HTML canvas, React Flow and ELK for diagrams, KaTeX for notation, XState for cue and session lifecycles, Fastify with WebSockets, Yjs for collaboration, Playwright and Vitest for tests, and one Docker image for hosting with a Nebius Serverless AI Endpoint path prepared.

Challenges we ran into

• Speaking continuously versus model latency. A final transcript arrives about 1.1–1.4 s after the last word, Nano routing takes 0.6–2.7 s and Super 1.1–2 s. Our first scheduler cancelled whatever was being interpreted as soon as the next phrase started, so a presenter who kept talking silently lost edits. We measured it end to end (a spoken "Highlight revenue" took 4.9 s to appear) and rebuilt the scheduler: finals queue in order and are never cancelled by continued speech, interim speech is read by local rules first so emphasis appears in under half a second, and Nemotron confirms or corrects when the sentence ends.
• Riva finals are audio portions, not sentences. We keep a finalised prefix and retract speculative emphasis when an interim transcript is revised.
• Strict JSON Schema in practice. Flat nullable objects decode reliably, but models still leave a kind's own fields null, so a normaliser drops half-filled steps before validation.
• A public demo without accounts. Project IDs are read capabilities and edit tokens are write capabilities; per-visitor and per-process limits sit in front of every paid call.

Accomplishments that we're proud of

• Nemotron decides every live cue, and the log shows which tier did what.
• Emphasis follows the voice in 0.3–0.5 s, and an edit followed immediately by narration still lands, measured on the real hosted ASR and Token Factory path.
• Model-authored lessons are compiled and validated server-side before a single object is drawn.
• 226 backend tests and 37 browser tests, with provider calls isolated so the suites never spend credits.

What we learned

Token Factory's strict JSON output and the Nemotron thinking switch make a three-tier design practical, but latency variance per call matters more than the median when a human is talking. Measuring speech-end-to-render honestly changed the architecture more than any prompt change did.

What's next for Vizli

• A public hosted URL on a Nebius Serverless AI Endpoint (script and steps are ready; not yet deployed).
• A real-speaker evaluation: 50–100 untested utterances from several people, reported with target accuracy and latency.
• Bounded contextual follow-ups to an authored lesson ("show what changes when the DNS answer is cached") and a clean audience view.

Built With

• docker
• elkjs
• fastify
• grpc
• katex
• nebius-token-factory
• nvidia-nemotron
• nvidia-riva
• playwright
• react
• react-flow
• typescript
• vite
• websockets
• xstate
• yjs
• zod