Closed Signing for Fire TV

Sign language interpretation you can move, resize, fade or hide — on Fire TV, from the broadcaster's own footage.

Comment

Inspiration

On television, sign language interpretation is burned into the picture. Someone else decides how large the interpreter is and where they sit, and the viewer can only take it or leave it. For many Deaf people a signed language is their first language and written English their second, so captions alone are not equal access — yet the interpreter they rely on is the one part of the screen they cannot control.

The ITU described the fix in 2013: deliver the interpreter as a separate video and let the receiver place it. European broadcasters piloted it over HbbTV a few years later; no consumer TV platform ships it. I wanted to see whether Vega could.

What it does

Closed Signing is a Vega OS player that takes existing interpreted broadcasts — here, the Scottish Parliament's own BSL coverage — and plays the programme and the interpreter as two separate layers. From the remote, while the video plays, the viewer moves the interpreter to any corner, resizes them, fades them or hides them. A first-run walkthrough asks once; the app remembers.

Captions run alongside, and never collide with the interpreter. The verbatim track is the Parliament's own Official Report — the edited record its reporters write — timed to the audio automatically and published unchanged. A simplified track in plain English is offered too, labelled on screen as written by AI and not checked by a person.

When the interpreter rests — during applause, between questions — the app can fade them a little and bring them back the moment signing resumes, driven by an activity track measured from the video.

Nothing is synthesised. Every sign comes from a human interpreter; the format makes that a required field.

How I built it

  • The player is React Native for Vega: two VideoPlayer surfaces decoding at real time (measured 0.99–1.01 playback rate on both, before and after the second starts), a layer and caption layout engine, D-pad overlays that adjust live video, persisted preferences, and the auto-dim, which runs on the device from the manifest's per-second activity track.
  • The format is the open Signed Track Manifest: a JSON manifest per title describing the programme layer, one or more sign tracks (with a required provenance, a public confidence, the crop and how it was made), caption tracks (with generatedBy and reviewed for machine text), and the activity track. Schemas, a validator and negative examples included.
  • The pipeline (Python, in the same repository) finds the interpreter with a person detector, classifies the layout, cuts the two layers with ffmpeg, measures hand and face motion per second, and builds captions from a publisher's transcript: speech recognition (Whisper locally, Amazon Transcribe on AWS) gives word timings only, a longest-common-subsequence alignment puts the Report's words on that clock — 88.7 % and 85.3 % of words anchored on the two sessions — and cues are cut at two lines of 42 characters.
  • AWS runs it: four CDK stacks (S3, ECR, Batch on Fargate, IAM), the image built by GitHub Actions over OIDC with no stored keys, and a full 32-minute session processed end to end on Batch in 29 minutes with Transcribe as the clock. Media for the app is served over HTTPS.
  • Tests: 385 across the three codebases (pipeline and format 185, app 185, infrastructure 15), with CI on both repositories.

Challenges I ran into

Fifteen of them are written up as a friction log, each with steps, the exact error and the workaround. The ones that shaped the app: Back, Select and Menu are never delivered to a Vega app; opacity on a View does not reach the video plane (the fade is a scrim drawn over it instead); the bundled AsyncStorage silently does nothing; KeplerVideoView ignores its own position and size. On the AWS side the account began in a guardrail organisation that denied CloudFormation, Batch and ECR, and Bedrock refused Anthropic model access outright — so the simplified tracks in the catalogue were written by hand and labelled as such.

The hardest technical problem was bounding the interpreter automatically. Detection is solved — 14 of 14 real broadcasts found, confidence 0.96–1.00 — but the crop's overlap with hand-measured ground truth is 0.727, not good enough to publish unattended. The shipped titles use a hand-measured rectangle, and the manifest says so (method: manual-crop).

Accomplishments I'm proud of

  • Two decoders at real time on Vega, verified, with the measurements published.
  • Captions that are the publisher's own words, timed by machine but written by people, with the alignment figures in every title's analysis.
  • An open format that makes "this is a human interpreter" a fact a player can check, and makes machine-written captions say so.
  • Being specific about what does not work yet.

What I learned

That the hard part of accessibility is rarely the feature; it is honesty about provenance — who made this, a person or a model, and has anyone checked it — and putting that on the screen, not in a file.

What's next

In the order I would do it, each step building on something that already exists.

1. Put it in front of the people it is for. Nobody who signs has used this yet, and that is the most important gap. Next is testing with Deaf viewers through Deaf-led organisations — which corner, which size, whether auto-dim helps or distracts, whether the simplified captions are welcome or patronising — and changing the defaults to what they say.

2. Have people check the AI-written captions. The format already has the step: a simplified track becomes reviewed: true only when a named person signs it off (Caption.mark_reviewed). The next step is a small review workflow so Deaf reviewers can correct and approve a track, and the badge on screen changes from "not checked by a person" to who checked it. When Bedrock access arrives, the pipeline writes the first draft; people still have the last word.

3. Find the interpreter without a hand-measured rectangle. Detection already works on every broadcast tried; the crop does not yet stop at the panel's edge. The idea queued next is a spatial-uniformity rule — stop growing the window when it leaves the flat backdrop behind the interpreter — measured against the same ground truth, and shipped only if it beats 0.727.

4. More publishers and more sign languages. The format already carries several sign tracks per title, each with its own language. Other parliaments and public broadcasters that interpret their output — in BSL, ASL and other sign languages — can be added from footage they already publish, with their transcripts as the captions where they have them.

5. Skip the extraction altogether. A broadcaster films the interpreter separately anyway. Exporting that feed as its own layer, with a manifest, removes the pipeline from the picture and gives a perfect crop for free. The open format is written so a publisher can do exactly that.

6. Live. The ITU's model is live by design: two streams, placed by the receiver. The player already plays HLS; a live event with the interpreter as a second stream is the same player, not a new one.

7. On real hardware, and in the Appstore. Everything is verified on the Vega Virtual Device. Next is a Fire TV device, a performance pass, and publishing the app so viewers can install it.

8. Wider access in the player. Screen-reader support checked on hardware (the controls already carry roles and values) for deafblind viewers and viewers with Usher syndrome; a viewer-adjustable timing offset for interpreters who run behind the speech (the format's syncOffsetMs, which the player already applies).

For the judges: the four criteria

Tech Implementation. Two simultaneous video surfaces on Vega, measured at real time; a layout engine that keeps captions clear of a movable interpreter; an on-device auto-dim computed from a per-second activity track; preferences that persist across launches. Behind it, an open format with JSON Schemas and a validator, and a pipeline that detects the interpreter, cuts the layers, measures signing activity and aligns a publisher's transcript to the audio — scored against hand-labelled ground truth, with the failures published alongside the successes. On AWS: CDK, Batch on Fargate, Transcribe, S3 and ECR, built and deployed for real, a full session processed in 29 minutes. 385 tests and fifteen friction-log entries.

Design. One focus language across the app; overlays that adjust the interpreter over live video rather than a paused frame; a first-run walkthrough where you place the interpreter before you start; captions that move out of the interpreter's way and lift over the transport bar; a title page that says which caption modes exist before you press Play; plain-sentence errors with one action; and provenance on screen wherever machine-written text appears.

Potential Impact. Deaf and hard-of-hearing viewers, for many of whom written English is a second language. In a study of 51 deaf and hard-of-hearing sign language users, adding captions to sign language interpreter videos raised comprehension by 24 % among deaf viewers and 42 % among hard-of-hearing viewers (PLOS ONE, 2015); this player is built so the two coexist. It needs no new filming: a publisher that already broadcasts with an interpreter can offer it from footage it holds, and the open format means any player can implement the same receiver. It maps to WCAG 2.2 success criterion 1.2.6 and to the ITU's closed-signing receiver model (see "Prior work").

Quality of the Idea. Closed signing as a receiver feature, the way closed captions are — a decade after the standard described it and before any TV platform shipped it. Two choices make it more than picture-in-picture: the interpreter is never synthesised and the format makes that a required, checkable field; and the captions are the publisher's own record, timed by machine but written by people.

Prior work

This problem has been named before; what has been missing is a shipping receiver.

  • ITU-T FG AVA, Technical Report Part 10 — Draft recommended requirements of TV receiver for closed signing (24 October 2013). The receiver model this player implements: programme and signer delivered separately, the receiver placing the signer. https://www.itu.int/pub/T-FG-AVA-2013-P10
  • Report ITU-R BT.2448 — Technical realisation of signing in digital television (2019; revised 2025). Describes closed signing by a separate signer stream mixed in the receiver, where "viewers can modify the position and the size of the signer", and cites user tests preferring a rather large signer at the lower right. https://www.itu.int/pub/R-REP-BT.2448
  • HBB4ALL (EU project, 2013–2016) — piloted closed signing over HbbTV with the broadcaster rbb: viewers chose size, position and layout of a pre-mixed signer window delivered over the internet, tested with sign language users. https://cordis.europa.eu/project/id/621014
  • Comprehension: adding captions to sign language interpreter video raised comprehension by 24 % (deaf) and 42 % (hard of hearing) in a study of 51 users. PLOS ONE, 2015. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0127577

Honest limits

  • The catalogue is small and institutional: two full sessions and two excerpts of Scottish Parliament coverage, plus a synthetic demonstration title.
  • A TV app cannot overlay other apps; this works on content delivered to it.
  • No testing with Deaf viewers yet.
  • The interpreter rectangle on the shipped titles is hand-measured.
  • Verified on the Vega Virtual Device, not on Fire TV hardware.
  • Amazon Bedrock was refused for the account; the simplified tracks were written by hand by an AI assistant and are labelled as unreviewed.

Built With

Share this project:

Updates

Submission history