SynapseTV - Spatial AI for Fire TV & AWS

Ambient Spatial AI on Fire TV combining edge perception with AWS Bedrock multi-agent swarm for contextual shopping, tactical sports explainers, and senior care emergency triage.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Inspiration

Living room television has remained fundamentally passive for decades: when you fall asleep, step into the kitchen, or get confused by intricate sports tactics or dialogue-dense films, the TV keeps playing into an empty void.

We built SynapseTV to transform Amazon Fire TV (Fire OS / Vega OS) from a passive display into an empathetic, context-aware Ambient Spatial Intelligence Hub. By synchronizing inward optical telemetry with outward video scene perception, SynapseTV proactively adapts the viewing experience in real-time.

What It Does

  • Dual Perception Pipeline: MediaPipe edge neural models process viewer attentiveness, confusion, and spatial hand gestures on-device. Zero raw video frames ever leave the room.
  • Proactive Living-Room Adaptations:
    • Auto-Pause & Smart Bookmark: Detects when a viewer falls asleep or steps away, micro-bookmarks the frame with a summary, and auto-resumes once focus returns.
    • Tactical Micro-Explainers: Automatically surfaces contextual breakdowns when complex plays or narrative shifts occur.
  • Direct Amazon Ecosystem Synergy:
    • Prime Video & Amazon Ads: Real-time in-stream item spotlighting with 1-click Fire TV remote (Enter key) to Add to Cart, plus mobile QR Amazon Pay checkout (yielding 3.2× CTR uplift).
    • Senior Care & One Medical Emergency Triage: Optical fall and unresponsiveness sentinel with a non-intrusive 30-second failsafe audio countdown before alerting Amazon One Medical and family caregivers via AWS SNS.
  • B2B Signage & Telemetry HUD: Displays frame ingestion rate, Bedrock compute load, and gaze retention analytics for commercial venue operators.

How We Built It

  • Fire TV 10-Foot Spatial UI: Built with Next.js, React, and Tailwind CSS, featuring high-contrast amber focus rings and strict 2D geometric D-pad graph routing matching physical Fire TV remote keycodes (19, 20, 21, 22, 23, 4, 179).
  • AWS Bedrock Cognitive Swarm: Multi-agent orchestration layer powered by Claude 3.5 Sonnet / Haiku with multi-tier fallback routers.
  • Backend Pipeline: Asynchronous FastAPI server serving bidirectional WebSocket streams and REST APIs.
  • Edge Privacy & Thermal Preservation: 100% local MediaPipe inference emitting solely numerical coordinates, supported by an adaptive FPS throttle (2 FPS Eco to 60 FPS Turbo) and a physical camera kill-switch.
  • Containerization & Cloud Infrastructure: Multi-stage production Docker Compose stack, AWS ECS Fargate readiness, and instant serverless deployment on Vercel.

Challenges We Overcame

  1. Fire TV WebView Keycode Normalization: Solved D-pad focus trapping and synthesized directional routing across differing Fire OS and Chromium webview remote implementations.
  2. Edge Hardware Preservation: Implemented an adaptive sampling engine throttling frame inference to 2 FPS when idle, eliminating thermal throttling on low-power Fire TV sticks.
  3. False-Positive Triage Guard: Created an empathetic 30-second cancellation buffer with prominent "I'm Just Relaxing" suppression to prevent false 911/emergency dispatches.

Accomplishments We're Proud Of

  • Successfully running inward edge vision models locally without transmitting a single pixel off-device.
  • Achieving a sub-second response loop between real-time Bedrock swarm reasoning and on-screen 10-foot overlays.
  • Architecting four concrete B2B and consumer monetization models across Amazon's existing retail, ads, and healthcare verticals.

What We Learned

  • Designing for the 10-foot experience requires entirely different interaction paradigms than web or mobile, where latency, high contrast, and predictable remote navigation are paramount.
  • Multi-agent fallback routing is essential for mission-critical living room AI to maintain zero user-perceived downtime.

What's Next for SynapseTV

  • Native Vega OS runtime package distribution via the Amazon Appstore.
  • Alexa Together integration for bidirectional acoustic living room emergency monitoring. ## Accomplishments that we're proud of

What we learned

What's next for SynapseTV - Spatial AI for Fire TV & AWS

Built With

Share this project:

Updates

Submission history