MIRA TV

Autonomous remote-control QA agent for Fire TV apps that navigates D-pad focus, detects unreachable elements and traps, and uses Amazon Bedrock AI to explain navigation defects.

Comment

Inspiration

Designing for the 10-foot television experience (Fire TV & Android TV) is completely different from mobile touchscreens or desktop mice. Users interact with applications entirely through a directional remote (D-pad: Up, Down, Left, Right, Select, Back).

Amazon’s official Fire TV Guidelines mandate three fundamental principles for TV usability:

  1. Reachable elements: Every interactive element on screen must be accessible via D-pad navigation.
  2. No focus traps: Users must always be able to navigate back or escape any view or component.
  3. Clear focus indicator: Every focused element must visually contrast against its unfocused state.

In modern TV development (such as React Native with spatial navigation), focus bugs are notoriously easy to introduce and frustrating to test manually across dozens of screens. We were inspired to build MIRA TV: an autonomous remote-control QA agent that tests Fire TV applications like a real user—driving D-pad input, mathematically mapping focus topologies, detecting critical accessibility defects, and explaining them with generative AI.

What it does

MIRA TV acts as an autonomous QA engineer holding a physical Fire TV remote:

  • Autonomous D-Pad Exploration: Drives the app over ADB on an Android TV emulator (or via synthetic fixtures), systematically traversing screens and building a directed navigation graph $\mathcal{G} = (V, E)$ where nodes $V$ are focusable elements and edges $E$ are D-pad directional transitions.
  • Deterministic Defect Detection: Unlike brittle AI testers, MIRA TV uses algorithmic gates to deterministically catch three critical defect types:
    • UNREACHABLE: Expected elements that cannot be reached by any D-pad path: $$V_{\text{unreachable}} = V_{\text{expected}} \setminus V_{\text{visited}}$$
    • TRAP: Controls where focus can enter, but reverse-reachability BFS confirms the user cannot escape back to an exit target: $$\text{Path}(v_{\text{trap}} \to V_{\text{exit}}) = \emptyset$$
    • NO_FOCUS_INDICATOR: Elements that gain focus but have no visible change, evaluated via bounding-box pixel delta analysis: $$\Delta_{\text{diff}} = \frac{1}{W \times H} \sum_{x,y} |I_{\text{focused}}(x,y) - I_{\text{unfocused}}(x,y)| < \tau_{\text{min}}$$
  • Amazon Bedrock AI Explanations: Once a defect is deterministically flagged, Amazon Bedrock (Claude Vision via the Converse API) analyzes the screenshots and logcat telemetry to write plain-language developer explanations and remediation guidance.
  • Verified Repair & Reporting: Applies unified patch diffs (.patch), re-explores the screen to verify resolution, and compiles a self-contained, offline HTML visual report contrasting defective states (Red) with verified fixes (Green).

How we built it

We implemented MIRA TV with a modular, contract-driven architecture:

  • Target Application: Amazon’s official react-native-multi-tv-app-sample running on Android TV (1080p, API 34), styled with react-tv-space-navigation.
  • Focus Telemetry Hook (Contract C1): A lightweight React Native hook (app/mira/reportFocus.ts) emitting structured MIRA_FOCUS telemetry over ADB logcat with element IDs and pixel bounds (active only in __DEV__ mode).
  • Core Engine (Python 3.11+):
    • Abstract Driver Layer: An AdbDriver for driving real emulators via subprocesses, keyevents, and screencaps, alongside a FixtureDriver for instant, offline replay without an emulator.
    • Graph State Machine: Traverses directional D-pad topologies, enforcing depth limits and cycle avoidance.
    • Amazon Bedrock Integration: Bedrock Converse API with local disk caching (ai_cache/) for fast, cost-effective reproducibility.
  • Visual Report (Contract C5): A single self-contained HTML/CSS dashboard that bundles interactive navigation graphs, screenshots, and Bedrock diagnostics with zero external CDNs or NPM dependencies.

Challenges we ran into

  1. Hardware Virtualization & Environment Isolation: Android TV emulators demand significant RAM and hardware virtualization (VT-x/Hyper-V). We abstracted our engine with FixtureDriver, allowing development, test suites, and CI workflows to run anywhere without launching an emulator.
  2. Preventing AI Hallucinations: AI models can hallucinate test verdicts. We established a strict architectural rule: Determinism decides; AI only explains. The graph algorithms determine pass/fail; Bedrock provides human insight.
  3. Spatial Navigation Dynamics: Real-time Euclidean distance calculations in TV focus engines can cause unexpected focus jumps during list virtualization and drawer transitions.
  4. Reliable Subprocess Timing: Handling ADB keyevents, screenshot capture, and logcat parsing on Windows required robust timeout guardrails and clean error handling.

Accomplishments that we're proud of

  • Full Autonomous Loop: Completed an end-to-end autonomous flow: Launch $\to$ Explore $\to$ Detect $\to$ AI Explain $\to$ Apply Patch $\to$ Re-verify Green $\to$ Generate Report.
  • 100% Deterministic Verdicts: Zero flakiness or false alarms in defect classification.
  • Ultra-Lean Tech Stack: Built the entire Python engine with only 3 dependencies (Pillow, boto3, pytest), implementing custom config parsers and schema validation without dependency bloat.
  • Enterprise-Grade Offline Reporting: Created a beautiful, standalone HTML report that works completely offline from file://.

What we learned

  • 10-Foot UI Accessibility: Gained deep appreciation for Amazon’s TV guidelines—accessibility traps that feel minor on web or mobile are catastrophic blockers on a TV remote.
  • Contract-First Teamwork: Defining strict JSON schemas (C1–C5) early made parallel development between app instrumentation and engine logic completely seamless.
  • Right Role for Generative AI: LLMs and vision models are best used for contextual translation, explanation, and developer advice, while algorithmic code should handle verification and gating.

What's next for MIRA TV

  • Open Source Contribution: Submit a pull request to Amazon's sample repository adding focus telemetry and automated CI verification.
  • Automated AI Remediation: Use Bedrock's code generation capabilities to synthesize and test .patch fixes automatically from navigation graphs.
  • Voice Remote & Alexa Intent Testing: Extend beyond D-pad exploration to test voice intents, deep-links, and search navigation on Fire TV.

Built With

Share this project:

Updates

Submission history