Hoot: the Storytime Companion

It didnt turn out how I expected
1st version correctly identifying my shirt with Nemotron Super
the 'final' form for now
HAT is cool
shake testing

JUDGES: PLEASE DO NOT JUDGE THIS PROJECT

I’m helping organize this hackathon and am an NVIDIA employee. I entered Hoot for fun, learning, and sharing. Not for judging or prizes.

Inspiration

Reading Chinese books with my children sometimes means stopping at an unfamiliar character. I’m building Hoot to help with those moments without taking us out of storytime, then turn difficult words into opportunities to practice later.

I also wanted to explore what a personal AI companion feels like when it has a physical presence: something we can speak to, show a book, pick up, and interact with.

What it does

Hoot is a working Jetson-based AI companion with English and Mandarin conversation, camera-based assistance, and a local reading dashboard.

It can capture book covers and pages, use vision to recognize their contents, and save pages in a shared local library. The dashboard supports editable recognized text, pinyin, saved vocabulary, and practice cards. An animated owl connects the physical companion to the web interface.

Hoot also understands its available hardware through Hermes tools. It can take a requested camera snapshot, read sensors, display LED expressions, adjust speaker volume, and switch speech modes. Activity lights show microphone, camera, and speaker use, while calibrated motion updates the dashboard’s representation of its body.

Following reading aloud, detecting hesitation, and automatically offering timely pronunciation help remain goals—not completed features.

How I’m building it

NVIDIA Nemotron 3.5 Super VL, served through Nebius Token Factory, powers Hoot’s language, vision, and tool decisions. Token Factory lets a small physical robot use a large multimodal model without hosting that model on the Jetson.

Hermes runs on the Jetson Orin Nano and supplies the agent harness, persistent conversation context, and tools. The Jetson also hosts the authenticated web app, local library, camera integration, USB audio, and Sense HAT controls.

Speech has two pathways:

Local: Whisper recognition and Kokoro speech synthesis.

Cloud: Gemini transcription and text-to-speech, with local fallback.

Gemini handles speech only; Hermes and Nemotron generate Hoot’s answers in both modes. Wake detection stays local, and idle operation avoids background model requests.

Books and study records currently stay on the Jetson. Nebius Object Storage synchronization is not implemented. Local speech and storage are available components, but general assistant reasoning still requires a connection to Nebius.

Challenges and testing

Latency is the biggest immediate challenge. Six English and Mandarin audio trials measured approximately 3.6–7.0 seconds from releasing the recording control to playback launch. Actual audible output follows that event; these are not acoustic onset measurements.

Microphone startup is now about a tenth of a second. Most remaining delay comes from transcription after release and waiting for the model’s answer. Subsecond response is a goal we have not achieved.

Chinese recognition, pronunciation quality, natural turn-taking, and reliable operation around room noise also need broader testing. Recognized page text remains editable rather than being treated as infallible.

Progress so far

Hoot has moved beyond the concept stage: the physical voice pipeline, Hermes/Nemotron integration, camera tools, reading dashboard, local library, hardware controls, and motion visualization are implemented.

The source is published on GitHub with setup instructions, privacy boundaries, and an English/Mandarin audio-test appendix. Current validation includes 67 server tests, 14 Python test files, browser checks, and a successful production build.

The project explores both Personal AI and Physical AI themes: persistent personal assistance combined with an agent that senses and responds through real hardware.

What’s next for Hoot

Next, I’m focusing on faster responses and validating the complete physical-book-to-vocabulary-review experience. Streaming transcription, faster first model output, and overlapping answer generation with speech are candidates for reducing the wait.

Reading alignment and hesitation-aware pronunciation assistance come after that foundation is reliable. Storage synchronization and offline model assistance remain future work.

The longer-term stretch goal is an owl-inspired wearable body with a camera eye, a mini-projector eye, and a swiveling, tilting head. Experimental body-design work has begun; motor control, projector integration, and wearable testing are still ahead.