JAQUARxFIRE

Seamless Fire TV control meets generative AI at the edge

Comment

Inspiration

We’ve all experienced the frustration of smart TV navigation: hunting for a misplaced physical remote between couch cushions, painstakingly clicking one letter at a time on an on-screen TV keyboard with arrow buttons, or asking built-in TV voice assistants a complex question only to get a generic search result or an unhelpful web link.

While televisions have evolved into brilliant 4K HDR displays, the input mechanism has remained stuck in the past. We asked ourselves: What if any television running Amazon Fire OS could turn any smartphone in the room into an instant, zero-friction tactile remote—and supercharge the living room viewing experience with the contextual generative intelligence of AWS Bedrock?

That sparked JAQUARxFIRE (FireBridge AI Ultra): a bridge between living room entertainment, real-time edge computing, and cloud foundation models.

What it does

JAQUARxFIRE transforms any television running Amazon Fire OS into an intelligent, multi-modal entertainment hub:

  1. Zero App Downloads (Instant QR Pairing): When launched, the Fire TV app displays a dynamic, high-resolution QR code inside a glowing neon frame. Any viewer simply points their iPhone or Android camera at the screen to instantly load a dark glassmorphic Progressive Web Remote in their browser—no app store downloads, Bluetooth pairing, or sign-ups required.
  2. Sub-5ms Duplex Control: Viewers can switch seamlessly between a Tactile D-Pad (with micro-haptic clicks), a Fluid Gesture Touchpad (with visual particle trails), volume rockers, and playback controls that execute instantaneously on Fire OS via Android KeyEvents.
  3. Mobile Voice Dictation: Utilizing the Web Speech API directly in the mobile browser, viewers can tap the pulsing cyan microphone button and dictate natural speech queries with real-time on-screen speech-to-text transcription.
  4. AWS Bedrock Ambient AI Assistant: One-tap contextual pills ("Who is this actor?", "Summarize last 10 minutes", "Explain plot twist", "Kid-Friendly content check") or custom voice questions invoke foundation models (Amazon Nova Lite and Anthropic Claude) on AWS Bedrock.
  5. Non-Intrusive Floating TV HUD: Rather than interrupting the viewer with full-screen takeovers, generative responses smoothly slide into the corner of the TV screen on a translucent, glassmorphic HUD card while simultaneously syncing back to the viewer's mobile remote.

How we built it

JAQUARxFIRE was engineered with an offline-first, dual-tier architecture bridging edge hardware with cloud foundation models:

  • Fire TV Host App:
    • Language & SDK: Native Kotlin built for Android SDK 34 / Fire OS.
    • Local Micro-Servers: Embedded Java-WebSocket and HTTP servers running on port 8080 directly on the television, listening for incoming mobile client connections.
    • QR Code Engine: Built with ZXing to dynamically encode the TV's local LAN IP address into an animated QR target.
    • Input Dispatcher: KeyCommandExecutor translates incoming JSON WebSocket payloads into native Android KeyEvent injections (KEYCODE_DPAD_UP, KEYCODE_ENTER, KEYCODE_VOLUME_UP, etc.).
    • Cloud Intelligence: Integrated with the official AWS SDK for Kotlin (aws.sdk.kotlin:bedrockruntime) to invoke Amazon Bedrock models (amazon.nova-lite-v1:0 and anthropic.claude-3-5-sonnet-20241022-v2:0).
  • Mobile Web Remote (Client):
    • Built using pure HTML5, modern Vanilla CSS (glassmorphism/flexbox/grid), and lightweight asynchronous JavaScript.
    • Interfaces with the mobile browser's Web Speech API for on-device voice transcription and the Vibration API for tactile haptic feedback.
    • Connects over persistent bidirectional WebSockets for instantaneous command delivery.

Challenges we ran into

  1. Network Address Translation & Local Discovery: Fire TV sticks and smartphones on home Wi-Fi networks frequently switch between 2.4 GHz and 5 GHz bands or encounter AP client isolation. We resolved this by building automatic LAN interface detection into Kotlin's NetworkUtils and implementing an embedded HTTP server directly on the TV so the phone doesn't rely on external discovery protocols like mDNS/Bonjour.
  2. Key Injection Security in Android/Fire OS: Fire OS imposes strict security boundaries on background key injection. We implemented an Instrumentation-backed executor and paired it with system audio manager hooks to guarantee reliable volume and navigational control across all views.
  3. Low-Latency Streaming AI Responses on TV: Displaying streaming LLM tokens on a 10-foot television interface without causing UI frame drops or stutter required careful thread management. We decoupled the AWS Bedrock network invocation onto Kotlin coroutine Dispatchers.IO and used hardware-accelerated XML slide-in animations on the main UI thread.
  4. Resilient Hackathon Fallbacks: To ensure seamless demonstrations regardless of AWS credential expiration or connectivity limits, we architected an intelligent dual-mode fallback inside BedrockService that gracefully switches between live Bedrock endpoints and rich contextual simulations.

Accomplishments that we're proud of

  • Zero-Friction UX: Anyone entering the room can participate in controlling the TV and querying AI in under 3 seconds just by scanning the TV screen.
  • Near-Zero Latency: Commands sent from the smartphone execute on the TV in under 5 milliseconds over local WebSockets.
  • Cinematic Ambient Design: Designed a tailored 10-foot TV UI featuring glassmorphism cards, glowing neon status badges, and non-disruptive ambient HUD overlays that feel like a native, premium entertainment operating system.
  • True Foundation Model Integration: Seamlessly incorporated Amazon's latest Amazon Nova Lite and Claude foundation models on AWS Bedrock directly into an embedded television ecosystem.

What we learned

  • How to architect bidirectional, multi-client WebSocket protocols for real-time edge devices using Kotlin coroutines.
  • The nuances of building for Amazon Fire OS and the 10-foot user experience (D-pad focus states, viewing distances, and readable contrast ratios).
  • Effective prompt engineering for living room AI—tailoring prompts to generate structured, spoiler-free cinematic insights that viewers can digest in 5 seconds without pausing their content.
  • Harnessing the AWS SDK for Kotlin for serverless generative AI orchestration directly on client devices.

What's next for JAQUARxFIRE

  • Multimodal Real-Time Video Analysis: Taking advantage of Amazon Nova's multimodal capabilities by capturing real-time paused screen frames and asking Bedrock: "Explain what happened in this exact frame."
  • Multi-User Living Room Voting: Allowing multiple family members or friends in the same room to connect simultaneously and vote on movie recommendations or queue media via their individual mobile remotes.
  • Smart Home Lighting Sync: Integrating AWS IoT Core to automatically dim living room lights when an AI query asks to "Set movie night vibe."
  • Personalized Viewing Profiles: Remembering individual viewer preferences and watch histories to suggest tailored content via generative AI dialog.

Built With

  • amazon-nova
  • android
  • android-viewbinding
  • aws-bedrock
  • aws-sdk-for-kotlin
  • claude
  • css3
  • fire-os
  • fire-tv
  • generative-ai
  • html5
  • iot
  • java-websocket
  • javascript
  • kotlin
  • kotlin-coroutines
  • kotlin-serialization
  • llm
  • material-design
  • qr-code
  • smart-tv
  • web-speech-api
  • websockets
  • zxing
Share this project:

Updates

Submission history