Magic Words - Storytelling through Captions

Track: Accessibility & Inclusion — Make Disney+ more accessible and inclusive for everyone, from any standpoint — physical, sensory, cognitive, linguistic, economic, cultural, generational, or otherwise.

Inspiration

Disney's real competitive edge has never been catalog size — it's emotional craft. The gasp at the reveal, the lump in your throat at the goodbye, the chill as the score swells. That's engineered, frame by frame, line by line.

But that craft doesn't reach everyone equally. A hearing viewer gets tone of voice, a tremor in a whisper, a swelling score, a flinch at a scream — all doing enormous emotional work underneath the dialogue. A caption-only viewer gets the exact same words, in uniform white text, same font, same size, no matter what's happening emotionally on screen. Two people can watch the identical scene and walk away with measurably different emotional experiences of the same story, and because it's invisible to most of the audience, it's never really been pressure-tested.

That gap — and the fact that it's mainstream, not a niche edge case (92% of mobile video is watched with the sound off, viewers are 80% more likely to finish a video when captions are on, and 37.5M+ Americans report some difficulty hearing) — is what got us building.

What it does

Magic Words is a captioning system that carries feeling, not just information. Instead of flat white text, captions shift in color, weight, and motion to match the emotional intensity of the line — a whispered secret renders differently than a scream, text gets heavier as a moment intensifies, motion mirrors the pacing of the scene. The words themselves never change; nothing about meaning is altered. The feeling just isn't stripped out in transit anymore. And it's fully optional — viewers can switch it on or off from the subtitle settings menu, the same way they'd adjust caption size or color today.

How we built it

We built a real classification skill on Claude that reads dialogue and tags each line with emotion, intensity, and delivery. For the demo, we pulled dialogue, timing, and speaker data from three internal repos, HiVE, Solo, and Disney (the Web Player repo), into one hardcoded JSON, then ran that actual Claude skill against it rather than simulating the output.

On top of those tags, we designed a full styling system: a font pairing (a condensed display font for emotionally charged lines, a calmer sans for quiet ones), a restrained color palette tied to emotional categories instead of a decorative rainbow, and thirteen distinct motion presets — a sharp "snap" for commands, a light jitter for urgency, a full shake for danger, a slow recede for ambient sound fading into the distance, and one bespoke "freeze and shatter" effect built specifically for a single dramatic beat. We burned this styling into a real exported video (The Incredibles' jungle-ambush scene) using ASS subtitles rendered through ffmpeg, and separately translated the whole system into CSS keyframe animations so it can hand off cleanly to engineering.

We also mapped out the production path: a new subtitle file lands in S3 and triggers EventBridge, Claude via Bedrock classifies it, the result is stored in S3 and served from CloudFront, and the Web Player passes it to a HiVE-style renderer that styles each phrase live during playback — all on infrastructure Disney already runs. We even ran the cost math: roughly $0.50–$1 in Claude/Bedrock inference per title classified, with the surrounding AWS plumbing adding only a few cents on top at scale.

Challenges we ran into

Getting caption timing frame-accurate was harder than expected — transcribing the whole scene at once kept blurring adjacent, overlapping, and interrupted lines together and drifting the timestamps. We ended up isolating narrow clips of each individual utterance and transcribing them one at a time, which was dramatically more accurate.

We also had to fight our own instinct to over-style. Early passes put color and motion on nearly every line and it read as busy and gimmicky rather than expressive. The version that actually landed reserved color and motion for genuine emotional peaks and kept most lines quiet — restraint turned out to be the harder design problem, not more effects.

On the technical side, we discovered our source recording had accidentally duplicated its opening few seconds (a rewind during screen capture), and had to detect and remove that cleanly — via audio cross-correlation — without disrupting the rest of the scene. And since we didn't have live access to the actual HiVE renderer during the hackathon, we had to build a faithful stand-in ourselves that could still prove the concept convincingly on a real video, real audio, and real dialogue.

What we learned

The biggest lesson was that subtlety is the feature, not a limitation of it — the goal isn't to make captions louder, it's to make the right moments louder while everything else stays quiet, which is exactly how emotional pacing works in the underlying performance. We also learned that making an accessibility feature optional isn't a compromise; it's what makes people trust it enough to actually use it. And practically: isolate-and-transcribe beats whole-file transcription every time when timing precision matters.

What's next

Richer emotion — moving from broad categories to nuanced intensity and mixed emotions. Personalization — user-adjustable intensity, reduced-motion, and color-blind-friendly modes. And full automation — scaling from one hand-built demo scene to a transcript-driven pipeline running across the whole catalog.

REPOS: https://github.twdcgrid.net/KUMAJ039/Disney-Captioning

https://github.twdcgrid.net/KUMAJ039/Disney-Streaming-NYC-Hackathon

https://github.twdcgrid.net/KUMAJ039/Disney-Web-Player