Inspiration
A good tutor doesn't wait for you to say "I'm lost." They hear it: the long pause before you answer, the "yeah, I think so" that means no, the voice that gets flatter every problem. AI tutoring tools or chatbots today only read the words, so they cheer you on right up to the moment you quit. We wanted a tutor that notices what a person would notice, and that you can talk to with your hands busy on a page of math, the way real studying works.
What it does
Ilumina is a real-time voice tutor for math.
- Less typing, more writing. A natural voice conversation with first words in about 1.5 to 2 seconds, powered by Deepgram.
- No more emotionless "absolutely! You were right." Every turn, Ilumina measures your speaking pace, your longest pause and how long you waited before answering, and Gemini's Decision 1 scores the audio for confusion, frustration, confidence and engagement, each with how sure it is.
- Adaptation. One odd turn is a blip and gets ignored. Two in a row, or one very confident reading, is a trend, and the tutor changes its teaching move: a smaller hint, a new explanation, a worked example, a harder problem or a check-in.
- It sees your work in real time. A GoPro (or any webcam e.g. Meta Raybans) shows Ilumina your page. It reads your handwriting and points at the exact step instead of you taking pictures of your work to receive feedback.
- Fell asleep during lecture? No problem. With a Vexo band on your wrist, Ilumina listens to a lecture, writes study notes (key ideas, sticking points, check questions), makes a study plan, and opens your next session from that class. Lecture review mode quizzes you on it; we used MIT 18.01 Lecture 3 for our demo.
- A live dashboard shows every real model output: the per-turn reading and confidence, the smoothed state, the voice evidence, the teaching move, a mic meter, and whose turn it is.
How we built it
- Pipecat runs the whole thing as one pipeline on a laptop, with a browser client over SmallWebRTC.
- Deepgram Flux does speech to text and turn detection, and its word timings feed our voice evidence (pace, longest pause, wait before answering). Deepgram Aura-2 is Ilumina's voice.
- Gemini 4 Hydrogen is the tutor brain. We tried Gemini 4 Argon, which thinks deeper, but its first words took over 4 seconds; Hydrogen gets there in about 1.5 to 2.
- Gemini Decision 1 scores each student turn from the audio plus the voice evidence, against rubrics for confusion, frustration, confidence and engagement, with a confidence value for each.
- A plain Python state tracker smooths those scores across turns into blip or trend, with no model calls, so it's cheap and predictable.
- Camera: a GoPro over USB, read with FFmpeg, hands the newest frame to Gemini with each turn. A C270 webcam or the browser camera works too.
- Vexo: a Vexo band records the lecture and streams it over Bluetooth through the Vexo open kit's band library (Opus, 16 kHz, 20 ms packets). If the band isn't available, the Mac microphone goes through the kit's simulated band on the same protocol, so nothing else in the pipeline changes.
- The dashboard is plain HTML and JavaScript fed by Pipecat RTVI server messages.
Challenges we ran into
- Speed versus depth. The deeper Gemini model gave better answers but a 4-second pause feels broken in a voice conversation. We switched to Hydrogen and kept emotion scoring off the critical path.
- Not overreacting. A tutor that changes course on every reading feels jumpy, so we built the blip-versus-trend tracker before wiring in adaptation.
- Seeing handwriting. A head-worn camera designed with one wide focal-length sees the page at an angle and from a distance.
- Filler words disappear. Deepgram Flux drops "um" and "uh" from transcripts, so pauses and wait time carry the hesitation signal instead.
Accomplishments that we're proud of
- First words in about 1.3 to 2.1 seconds, measured live, while scoring emotion on every turn.
- A tutor that really reads a handwritten calculus problem from a camera on your head.
- Every number on the dashboard is a real model output, not a mock.
What we learned
- How someone says something (intonation, change in pitch, etc) carries information the words don't, and word timings alone (pauses, wait time) are a strong signal.
- Smoothing matters as much as scoring: a tutor that reacts to every reading feels anxious, not caring.
What's next for Ilumina
- Wear the Vexo band all day, so Ilumina can follow a whole school day of lectures.
- Learn each student's own baseline, so a naturally slow talker isn't read as confused.
- Track progress across sessions and focus study plans on the moments a student struggled.