Inspiration
Every night, millions of people pause their TV, pull out their phones, and leave the story.
A kid asks "why is the sky blue?" during a documentary. The couch argues over a foul. A news claim flashes by unchecked. Halfway through a film, nobody knows who that guy is.
The answers exist, but reaching for a phone breaks the moment. We wanted the TV itself to talk back: one gesture, the answer on screen, and you never leave the story.
What it does
SPOTLIGHT is a curiosity remote for Fire TV. A 10-foot TV app pairs with a phone remote: hold push-to-talk, ask out loud, and the TV answers out loud. Every demo beat follows the same loop: confusion → ask → answer.
| Beat | You ask | You get |
|---|---|---|
| 🧒 Kids | "Why is the sky blue?" | A warm, read-aloud explainer |
| 🎬 Movies | "Who is that man?" | "Cary Grant, as Peter Joshua" in Charade (1963) |
| ⚽ Sports | "Why was that a foul?" | A replay description plus a one-line rule |
| 📰 News | "Is that claim true?" | A fact-check verdict with sources |
Voice commands like pause, rewind 30 seconds, and volume up work through the same push-to-talk button.
The core idea is TV self-awareness. Every 5 seconds, the TV reports its title, timestamp, and a video frame. Every question automatically carries what's on screen. There's no setup, no typing, and no photographing your TV.
Underneath, each domain has its own knowledge engine with curated answers, an LLM fallback chain for unscripted questions, spoiler-aware prompts, and an answer cache.
Honest boundary: this awareness covers content playing inside our TV page. Seeing into other apps needs native platform access that only Amazon can grant, and that is exactly the pitch.
How we built it
- Backend and UI: FastAPI + vanilla JS.
/tvis the 10-foot UI,/remoteis the phone remote with push-to-talk, and/demois a side-by-side view. - Voice loop: faster-whisper (speech-to-text) → ask pipeline → Piper TTS on the phone, plus speech on the TV. A keyword intent router splits device commands from questions, so "pause" never waits on an LLM.
- TV self-awareness: canvas frame capture + title + timestamp →
POST /api/tv/context→ auto-attached to every question. Explicit context always wins. - Answers: 8 knowledge engines with curated canonical answers first, then an LLM fallback chain (Amazon Bedrock → OpenAI → Ollama) for anything unscripted.
- Real-time sync: a WebSocket hub bridges phone and TV, and the TV shows a live voice-status pill (listening → thinking → speaking).
- Quality: 91/91 tests passing. Before submitting, we ran a structured AI-mentor review and fixed both P0 findings it raised.
Challenges we ran into
- The Vega simulator was a dead end. We couldn't get a WebView app running on the virtual device on our development machine, and we checked three different ways before moving on. The demo runs on a cloud Fire TV via Live Device Interaction instead.
- Our coding agent hit a quota wall mid-build. We built the improvement round directly through the API instead.
- A free ngrok tunnel kept dropping. The recording session needed three tunnel rebuilds.
- Cross-origin video blocks frame capture. We added a graceful fallback to title and timestamp metadata.
Accomplishments that we're proud of
- A working, verified curiosity remote, not a mockup: real public-domain film footage (Charade, 1963), a full voice loop, and TV self-awareness, all running on an actual Fire TV.
- Honest by design. The curated-vs-live split is disclosed, untested integrations are labeled, and stubs are named as stubs.
- Idea discipline. We ran 37 AI-mentor reviews across 5 rounds and killed four ideas, with the reasons documented.
What we learned
- On a TV, latency and fallbacks matter more than model quality. Every feature needs a plan B, such as metadata when a frame is blocked.
- Title plus timestamp gets you surprisingly far. The video frame makes answers sharper but isn't required.
- Saying plainly what's curated and what's live makes a prototype more credible, not less.
What's next for SPOTLIGHT
SPOTLIGHT is a prototype of something the platform should own. It needs three things from Amazon:
- Wake word on the remote, so no button is needed
- Frame-buffer access in every app, not just ours
- Alexa as the voice layer
Then it's not a demo. It's a Fire TV feature.
Built With
- amazon-bedrock
- fastapi
- faster-whisper
- fire-tv
- google-fact-check-api
- javascript
- ollama
- piper-tts
- python
- web-speech-api
