Every Seat

Fire TV that knows what's coming in a film before it happens, and softens, warns or describes each moment for whoever is on the couch tonight.

Comment

Inspiration

Movie night is a group activity, but comfort is personal. One person on the couch is bothered by strobing lights, another by sudden loud bangs, a kid wants a heads-up before the scary bit, and Grandpa can't see the screen well. Today the only tools are the volume button, closing your eyes, or not watching together.

Captions solved a similar problem decades ago: a small sidecar file that travels with a film and lets any player adapt it for the viewer. I asked: what if comfort worked the same way? So I built two things: Every Seat, a Fire TV app for the whole couch, and SenseVTT, the open format and player library it runs on.

What it does

Every Seat is a Fire TV app (Vega OS) that plays a film already knowing where the flashes, sudden loud moments and scares are, then adapts each moment for everyone on the couch at once.

  1. "Who's on the couch?" Pick everyone watching with the remote, or just tell Alexa+ through Every Seat's add-on ("Mom's joining Leo on the couch tonight"). Every Seat combines everyone's comfort settings into one couch lens, using the most careful choice of each kind. Set up your household once, like streaming profiles, with the remote or by voice; a demo household is there if you just want to try it.
  2. Flashes are softened automatically. Strobing sequences dim smoothly, starting before the first flash and releasing after the last, with a quiet "Flash softened" badge. I measured this with EA's open-source IRIS flash analyzer: a real failing strobe sequence in one of the demo films goes from FAIL (luminance swing 0.33) to PASS (0.071) with the app's 50% dim applied at its planned timing (an offline render of the same dim).
  3. Sudden loud moments are capped. When anyone on the couch wants it, Every Seat plays a comfort audio track that makes the jump at a real on-screen crash 18 dB smaller. It follows the couch: change who's watching mid-film and the track switches over on the fly.
  4. Scares and big bangs get a heads-up. Five seconds before a scare or a sudden loud moment, a non-modal card offers Soften (a live −18 dB volume duck plus a light dim), Watch (no change) or Tell me (a one-line, spoiler-light hint). No answer means Soften. It never pauses the film or steals focus.
  5. Audio description on your own phone. A blind or low-vision viewer opens Every Seat's page on their phone and taps Start once. From then on the phone follows the TV live: short spoken descriptions of what's on screen play in the film's quiet gaps, and they pause, resume and jump along with the film. Listen through the phone's speaker or any earphones; the rest of the couch hears the film as usual. The page labels every description as AI-generated.
  6. A heat-map seek bar shows where the marked moments are for this couch, before you press play.
  7. An Alexa+ add-on. Every Seat's voice companion is an MCP server built for Alexa+. Ask "Is this film okay for the couch tonight?" and the answer comes back as an interactive MCP Apps card: a timeline of the film's moments and how each will be handled for this couch. The demo runs it on a simulated Alexa+ screen, where Amazon Nova 2 Lite turns each spoken sentence into tool calls.

Built for the couch, not a settings menu. Everything works from the remote, every adaptation happens on its own, and the only choice offered during a film (Soften, Watch or Tell me) is one press that defaults to the gentle option. A first-run notice says plainly that Every Seat offers comfort, is not a medical device, and that the analysis can miss moments.

Under the hood: SenseVTT, "captions for the senses." I designed the cue format, built on WebVTT, and published it as open source with a validator, a detectors-only analyzer and a React Native player library with adapters for Vega, react-native-video and the web (github.com/zion-off/sensevtt, npm sensevtt). Every moment above is a SenseVTT cue, and Every Seat's playback runs on the library.

How I built it

On the TV (Vega OS, React Native, Vega Virtual Device).

  • Playback with the W3C Media API and Shaka (HLS, fMP4, 1 s segments) and two audio renditions per title (main and comfort).
  • Built on Vega's own pieces: the Kepler UI Components seek bar draws the heat map, FocusManager lets Up reach a heads-up card and hands focus back when it closes, useTVEventHandler drives the remote, announceForAccessibility reads every change aloud, deep links open a film for named people at a set time, and the household persists in AsyncStorage.
  • SenseVTT's scheduler and Vega adapter drive every dim and duck; Every Seat adds the couch lens, the heads-up cards and the comfort-track choice on top.
  • An event-anchored media clock: I measured that the reported position can run up to 0.66 s ahead of the picture right after play() and can jump about 3 s after a seek, so the clock freezes until playing/seeked and ignores jumps.
  • Dim and duck timing tuned from frame-by-frame screen recordings of the virtual device: dim edges land within about ±20 ms of the plan; the duck reaches −17.7 dB on schedule.
  • Remote-first design for the 10-foot screen: clear focus states, non-modal cards, a consistent design language and accessibility labels on every control.

In the cloud (AWS, us-east-1), pay-per-use.

  • Uploading a film to S3 starts the analysis automatically: EventBridge → Lambda → a job on Amazon Bedrock AgentCore Runtime (arm64 container, async sessions, traces in CloudWatch).
  • Inside, detectors measure and the model judges: ffmpeg detectors find loudness onsets and shot cuts, EA IRIS (the flash analyzer used in games) runs as an x86 Fargate task, and a Strands agent on Amazon Nova Pro (us.amazon.nova-pro-v1:0) reviews contact sheets and detector readings, decides which moments matter, rates intensity and writes spoiler-light previews. Pipeline timestamps come from the detectors, never from the model.
  • Output: a SenseVTT cue file and a comfort-audio recipe, written back to S3. Verified end to end on AWS: a 90 s excerpt costs about $0.03 of Nova.
  • Audio description: Nova Pro writes each description to a word budget for its quiet gap, from contact sheets plus a plot synopsis, and Amazon Polly speaks it. The pipeline measures every clip and sends any line that runs long back to Nova to shorten, so each one fits its gap. Repeats and camera jargon are rejected in code.
  • Infrastructure as code with the AWS CDK (Python) and the AgentCore CLI; default VPC public subnets, no NAT.

The Alexa+ add-on: an MCP server that drives the TV. A Node server speaking MCP 2025-11-25 over Streamable HTTP (the protocol version and transport Alexa+ add-ons use) exposes three tools: check_title_comfort (is this film okay for tonight's couch?), set_tonight_lens (who's watching) and remember_preference (save someone's comfort settings for every night). check_title_comfort returns an MCP Apps card: an interactive timeline of the film's moments for this couch. Tool handlers answer in a few milliseconds, well inside Alexa+'s 500 ms limit. Couch changes travel both ways over a WebSocket relay (newline framing, heartbeats, replay for late joiners): a tool call updates the TV live, and a couch picked with the remote reaches the companion, so the remote and voice always agree on who's watching; the household lives in a store that survives restarts. For the demo, a simulated Alexa+ screen stands in for the assistant: a conversation endpoint on Amazon Nova 2 Lite (us.amazon.nova-2-lite-v1:0) turns a spoken or typed sentence into those tool calls. Because it's plain MCP, the same tools can also plug into any other MCP-capable assistant.

The phone page: audio description in step with the TV. The TV sends its playback position over the same relay on every play, pause and seek, and every 2 s, stamped in the relay's time. The phone measures its own clock offset to the relay with round-trip probes, so neither device has to trust its own clock (the virtual TV's runs about 950 ms behind), and schedules each clip ahead on the Web Audio clock. In my measurements against the virtual TV, clips started within about 15 ms of their cue.

Challenges I ran into

  • Late is worse than nothing. My first dim test on a real strobe made IRIS results worse, because a dim that starts mid-strobe adds a transition. That finding shaped the whole design: the dim has to begin before the first flash, so the app needs to know what's coming; it ramps with wide margins and anchors its clock to player events.
  • Giving the model the right job. Comfort timing needs frame accuracy, and video models sample frames about once a second, so asking for timestamps from raw video was the wrong question. The fix was a division of labour: detectors measure every timestamp, and Nova Pro judges what matters from contact sheets and detector readings. That's where it shone: in my evaluation it found 8 of 9 events, against 6 of 9 for Nova 2 Lite.
  • Rules belong in code. Hard limits (loudness thresholds, word budgets, no repeated descriptions) are enforced by the tools, which send any rejection back to Nova Pro to revise. The model stays free to judge and write; the code guarantees the numbers.
  • The virtual device's quirks. Large WebSocket messages arrive split, seeking while paused doesn't repaint the picture, and switching audio languages freezes it for about 1.1 s. Each one is measured and worked around (36 entries in my friction log).
  • Narration that fits. First drafts of audio description repeated themselves and talked over several shots at once. Gaps, word budgets, measured clip lengths and shot-anchored sentences fixed it.

Accomplishments I'm proud of

  • A measurable comfort result: IRIS FAIL → PASS on a real film sequence, plus an 18 dB smaller jump at a real on-screen crash.
  • Remote, voice and phone working as one: tell the Alexa+ add-on who's joining and the TV's couch changes live, answers come back as an MCP Apps card, and the phone follows whatever the TV is playing.
  • Audio description that stays in sync on a real phone through pause and seek.
  • SenseVTT published on GitHub and npm for any player to adopt, with Every Seat as its first user.

Why it matters

People already go out of their way for this. Cinemas run sensory-friendly screenings with the lights up and the sound down, and viewers pay for timestamped scene alerts online just to know when a moment is coming. At home the tools are all-or-nothing: an audio description track plays for the whole room, and turning the volume down for one person spoils the film for everyone else. Every Seat brings that care into the living room, for each person, without pausing anyone's film, so nobody has to sit it out.

It reaches beyond one app because the cues travel with the film in their own file: any Fire TV app can adopt them with npm install sensevtt and the Vega adapter, and a platform could offer them on every title the way it offers captions today. Comfort stops being one person's problem and becomes part of movie night.

What I learned

Timing matters more than intelligence: the hard part of comfort isn't detecting a moment, it's acting on it at the right millisecond, for several people at once, without interrupting anyone. And AI is best used here as a judge over measurements, not as the measuring instrument.

What's next

  • Run on real Fire TV hardware and tune the measured offsets per device.
  • Take the add-on live on Alexa+ through the MCP Toolkit, with a hosted endpoint and OAuth account linking.
  • Analysis on the fly for titles that don't have a cue file yet: run the detectors ahead of the playhead as the stream buffers, then cache the finished cue file for everyone who watches next.
  • Publish SenseVTT 1.0 and invite feedback from accessibility groups, studios and player teams.
  • Human review of cues for studios that want to check them before release.
  • More comfort types (rapid camera motion, specific sounds) and more description voices and languages.

Built With

Share this project:

Updates

Submission history