Nod & Read

Inspiration

Comics are one of the most visual kinds of storytelling, and that makes them one of the least accessible. A blind reader can listen to an audiobook, but there's almost no way to listen to a comic. Dyslexic readers struggle with dense speech bubbles, and anyone who can't easily use their hands has to fight with tiny page-turn buttons. We wanted a comic to feel like a show you watch, one you can control just by moving your head.

What it does

Nod & Read turns comic pages into an accessible, hands-free experience:

🎬 One panel per screen. It finds every panel, puts them in reading order, and a camera glides across the page from panel to panel, like a motion comic.
🗣️ A voice for every character. Each character gets their own voice, with emotion ("[whispers]", "[urgently]"), plus sound effects under each panel.
🎧 Audiobook mode. A narrator describes the setting, the action, and each character's expression and body language.
🙂 Face control. Turn your head right for yes / next panel, left for no / back, close your eyes to toggle the narrator, and look up to pause. Face tracking runs entirely in your browser, so no video ever leaves your device.
❤️ A heart meter. Smiles and nods fill it, shakes drain it, and it picks what to watch next: more of what you liked, or a change of mood if you didn't enjoy it.
♿ Accessibility built in. Dyslexia-friendly and low-vision fonts, colour-blind correction applied to the artwork itself, emotion labels for readers who find expressions hard to read, full screen-reader and keyboard support, and translation into 7 languages, voiced by the same cast.

The library is curated: only public-domain and Creative Commons comics, like Little Nemo in Slumberland (1905) and Pepper&Carrot.

How we built it

Every page goes through an AI pipeline:

Magi v2 (Hugging Face) detects the panels. It was trained on manga, so we re-sort its right-to-left order into Western rows read left to right.
Google Gemini reads the whole page once to build a consistent cast list, then writes each panel's script in parallel: scene description, narration, dialogue with speakers and emotion tags, and sound-effect prompts. If one model is overloaded or rate-limited, requests fall back to another Gemini Flash model.
Gemini casts one ElevenLabs voice per character. ElevenLabs v3 performs the dialogue and generates the sound effects, and edge-tts voices the narrator in each language.
Everything is stored in Neon (Postgres plus object storage). Pages are cached by their SHA-256 hash, so nothing is generated twice.
The FastAPI backend streams pipeline progress over Server-Sent Events into a live visualization of the comic being read, which zooms seamlessly into the reader.

The front end is React 19 + TypeScript + Vite. MediaPipe Face Landmarker tracks 478 face points on-device, and three.js renders an animated 3D guide that teaches each gesture.

Head turn (yaw) comes from the face's rotation matrix:

$$\text{yaw} = \operatorname{atan2}!\left(-m_{8},\ \sqrt{m_{9}^2 + m_{10}^2}\right)$$

During setup you hold each turn until a bar fills, and we set your own threshold from how far you held it:

$$\theta = \operatorname{clamp}!\big(0.6 \cdot \operatorname{median}(\text{held}),\ 10^\circ,\ 24^\circ\big)$$

That way, people who can only turn a little can still use the whole site.

Challenges we ran into

Gestures are noisy. Head turns were misread as shakes, the "neutral" position drifted, and talking or another face walking into the frame set off false triggers. We added re-arming, grace periods, a stillness check, and auto-pause when you look away.
Winks didn't work for everyone. Many people can't wink with one eye, so we switched to closing both eyes for one second.
Too many gestures. We first had nods, shakes, gaze and winks all issuing commands. It was confusing, so we cut it down to two simple ones: right = yes, left = no. Nods now only fill the heart meter.
Left and right got flipped. The webcam image is mirrored, so we flipped head-turn direction more than once before confirming it on real hardware.
Old comics broke panel detection. Little Nemo's ornate 1905 layouts confused Magi, so we marked those panels by hand.
Keeping voices consistent across pages and separate uploads of the same comic took a whole-page cast list with stable character IDs.
Hosting cheaply. PyTorch is huge, so we made the server reading-only: it serves everything already stored, with no PyTorch and no model API keys, in about 1–2 GB of RAM.
Licensing. We dropped every comic we couldn't legally use, so the library is public domain and Creative Commons only.

Accomplishments that we're proud of

You can watch a full comic with character voices, sound effects and a narrator without touching anything.
Face tracking is private by design: it runs entirely in the browser.
Per-user calibration makes face control work for people with limited movement.
The live visualization shows the AI reading the comic, then zooms straight into the reader with no cut.

What we learned

Accessibility features help everyone. Captions, narration and hands-free control made the app better for all of us, not only the people we built it for.
Fewer, clearer gestures beat clever ones. Every gesture we removed made the app easier to use.
Testing on real webcams, real faces and real lighting catches bugs no simulation will.
Caching AI output by content hash keeps costs down and makes replays instant.

What's next for Nod & Read

A bigger library, through partnerships with public-domain archives and Creative Commons creators.
User testing with blind, low-vision and motor-impaired readers.
A mobile app with the front camera as the controller.
More languages and more expressive narration.
Letting creators submit their own licensed comics for review.