AccessFlow: Adaptive Navigation for Fire TV Accessibility

AccessFlow learns how you navigate Fire TV, detects where you struggle, and shortens the path — fewer presses and errors for users with motor impairments, adapted only with your approval.

Comment

Inspiration

Smart TV applications require complex menu navigation and repeated remote-control interactions. For users with motor impairments, every extra D-pad press is effort, and every overshoot is an error they have to undo. Fire TV already does a lot for vision accessibility (VoiceView, Text Banner, Adaptive Display), but navigation effort and motor difficulty are not adapted at all — the menu structure is fixed for everyone.

What it does

AccessFlow is an AI-assisted, self-adaptive navigation system for Amazon Fire TV that personalizes navigation based on individual interaction patterns:

  • Detects difficulty live — overshoots (passing a target and reversing), backtracks, wrong selects, direction reversals, and long hesitations are logged from real remote input.
  • Optimizes paths with a weighted graph — every screen and focus position is a graph node; remote actions are weighted edges. A Dijkstra search computes the lowest-effort path to the user's goal, and edge weights rise where this specific user struggles.
  • Provides contextual task recovery — when the system detects the user is lost, it offers the next step on the optimal path to the goal it infers.
  • Adapts only with user approval — adaptations (e.g., pinning a Subtitles shortcut on Home, larger focus targets) are suggested with the reason shown, and applied only when the user approves.

In the demo, the same task — turning on Subtitles — is completed before and after adaptation, and AccessFlow shows the measured improvement in presses and errors from the user's actual interaction.

How we built it

  • A Fire TV experience prototype (10-foot UI simulator) built as a mobile-first web app, so it can be used entirely by tapping an on-screen remote.
  • Navigation modelled as a weighted directed graph in JavaScript (44 nodes, 206 edges); Dijkstra's algorithm computes optimal paths and press counts live.
  • An interaction event logger that classifies difficulty events (overshoot, backtrack, wrong select, hesitation) in real time.
  • A suggestion/approval layer: no adaptation is ever applied silently.
  • Live metrics panel and a before/after comparison computed from the actual demo runs.

What's new in the combined build

  • Automatic shortcut discovery. AccessFlow ranks every destination by visit frequency × excess navigation cost (measured against the Dijkstra optimum) and proposes the shortcut itself — the proposal card cites the real numbers from your runs. A suggestion based on a single run labels itself "early — it sharpens as you navigate more."
  • Fitts's-law effort model. Edge costs combine distance and target size with a per-user error-rate multiplier learned live, so "effort" means more than raw press counts.
  • Movement profiles. Standard / Reduce horizontal / Reduce vertical user preferences reshape route costs by direction, so the optimum respects the user's own abilities (adopted from teammate Shivaprasad's parallel prototype).
  • Live path-effort tally. The metrics panel continuously shows the weighted cost of the path you are actually taking against the computed optimum.
  • Guided navigation. A step-by-step walkthrough ("Step k of n") watches your real presses and recomputes the route when you deviate (adopted from Shivaprasad's prototype).
  • Cross-session persistence. Run statistics, learned weights and your profile accumulate on-device across sessions — the discovery card cites "Across 2 sessions · 2 measured runs · 16 total presses." Nothing is uploaded, approvals are never auto-restored, and a Reset learning control is included.
  • Adaptation audit + undo. Every applied adaptation is logged with the evidence that triggered it, and each can be undone — consent you can reverse.
  • Goal prediction for recovery. When difficulty spikes mid-task, AccessFlow predicts your destination from your path so far and past visits — simple counting over the navigation graph, not an LLM, and labelled that way in the UI.
  • Fire TV app. The same engine ships as a native Fire OS app (WebView shell, source in the repo under fire-tv/, package com.arya.accessflow) so it runs on Fire TV with real remote key events.
  • Tested engineering. 20 automated tests (Dijkstra, Fitts model, discovery ranking, prediction, profiles, persistence, guidance) run in CI on every push.

User study

A structured user study (protocol in the repo under docs/study/) is in progress; the demo's measured runs demonstrate the mechanism, and the study will provide participant evidence.

Challenges we ran into

  • Modelling TV navigation as a graph so that "effort" is measurable, not just a feeling.
  • Detecting difficulty from raw D-pad input without annoying the user with false suggestions — solved with thresholds and user approval.
  • Building and testing the entire project from a mobile phone within the submission window.
  • On-device validation was blocked at the last step: Amazon's developer sign-in refused the available browser (an AI-agent block that persisted even when the user clicked sign-in himself), so the cloud Fire TV device farm could not be used for filming. The built Fire TV APK and its full wrapper source are in the repo, and the demo runs the real engine in the web prototype — all disclosed openly.

Accomplishments we're proud of

  • A real, working graph-optimization engine — not a scripted demo. Every number on screen is measured from live input.
  • The approval-first design: the system explains what it noticed and asks before changing anything.
  • A real, built Fire TV APK (package com.arya.accessflow, target SDK 34) with its full WebView-wrapper source in the repo — verifiable platform work, even though Amazon's developer sign-in blocked cloud device-farm testing on the final day.

What we learned

Accessibility is not only about seeing the screen — it's about the physical cost of reaching a goal. Measuring navigation as weighted effort makes that cost visible and fixable.

What's next for AccessFlow

  • A native Fire TV app using the same engine, with per-profile ability models.
  • Additional input modes (voice + switch control) feeding the same difficulty detector.
  • A structured participant study (protocol in the repo) to validate the mechanism beyond the demo runs.

Built With

Share this project:

Updates

Submission history