OmniLiving+: Autonomous Living Room Copilot for Alexa+

OmniLiving+ unifies Alexa+, Ring, Fire TV & Bee wearables via MCP Streamable HTTP and AWS Bedrock vision into an autonomous ambient living room copilot and home guardian.

Comment

Inspiration

Nowday's smart homes suffer from siloed intelligence:

  • Your Ring doorbell spots movement at the front door, but has zero semantic context on who is there or what they are doing.
  • Your Fire TV blares movie audio in the living room, oblivious to visitors or home safety alerts.
  • Your Bee wearable tracks vital signs, completely disconnected from the room environment.
  • Alexa waits passively on the counter for manual trigger phrases.

When Amazon announced the Developer Hackathon uniting Alexa+, Fire TV, Ring, and Bee for the first time, we asked:
What if Alexa+ could autonomously orchestrate all of these devices into a single home lab - customisable space, cohesive living room copilot?

By decoupling device silos with the open Model Context Protocol (MCP spec 2025-11-25+) over Streamable HTTP and pairing it with AWS Bedrock multimodal vision, we built OmniLiving+ — an intelligent ambient operating system that enables Alexa+ to actively sense, reason, and act across the entire home.

What it does

OmniLiving+ bridges Amazon's ecosystem into an unassailable multimodal experience across three core domains:

1. Context-Aware Doorstep Delivery (Ring + Fire TV + Alexa+)

When motion is detected at the front door while you are watching Fire TV:

  1. Ring Snapshot: The doorbell captures a high-resolution camera frame.
  2. Bedrock Multimodal Reasoning: AWS Bedrock (Claude 3.5 Sonnet / Amazon Nova) analyzes the frame in real time.
  3. Acoustic Audio Ducking: Fire TV automatically ducks media volume to ( 15\% ) so you never miss an alert.
  4. 1080p Picture-in-Picture Inset: Displays a subtle, non-disruptive live camera overlay in the upper-right corner with courier bounding box and carrier badge ("Amazon Prime Delivery").
  5. Spoken Chime: Alexa+ delivers a concise announcement ("Amazon Prime courier delivered your package at the porch"), then smoothly restores Fire TV media volume.

2. Proactive Family Care & Elderly Safety (Ring + Bee Wearable + Alexa+)

Unwitnessed falls on stairs and porches are a leading cause of senior injury:

  • If a senior resident remains stationary on the porch stairs for over 45 seconds, our Two-Frame Temporal Verification logic flags a standstill anomaly.
  • Alexa+ proactively speaks a gentle, non-alarmist check-in inquiry ("Are you feeling alright, Eleanor? Would you like me to notify family?").
  • Simultaneously, OmniLiving+ pulses a tactile haptic vibration alert on the resident's Bee wearable and synchronizes resting heart rate vitals (( 72\,\text{BPM} )) with the living room care console.

3. Perimeter Threat Hardening (Ring + Smart Deadbolt + Floodlights)

If an unidentified individual lingers after dark without approaching the doorbell:

  • AWS Bedrock classifies the scene as a perimeter threat (( \text{Confidence} \ge 96.0\% )).
  • OmniLiving+ dispatches an emergency MCP tool call: automatically locks the front deadbolt, elevates porch floodlights to ( 100\% ), and displays a security inset on the living room screen.

How we built it

       +------------------------------------+
       |   Alexa+ Ambient Intelligence      |
       +-----------------+------------------+
                         |
           Streamable HTTP / SSE (2025-11-25)
                         |
                         v
+--------------------------------------------------+
|        OmniLiving+ Self-Hosted MCP Server        |
|  [ring_inspect]  [bedrock_analyze]  [firetv_pip] |
|           [bee_sync]  [alexa_ambient]            |
+------------------------+-------------------------+
                         |
       +-----------+-----+-----+-----------+
       |           |           |           |
       v           v           v           v
  [Ring Pro]   [AWS Bedrock] [Fire TV]  [Bee Wearable]
  (Vision HUD)  (Claude CV)  (4K & PiP) (Vitals/Haptic)
  1. Self-Hosted MCP Server (Spec 2025-11-25+ Streamable HTTP):
    • Built in Node.js and TypeScript, exposing JSON-RPC 2.0 endpoints on POST /mcp and real-time Server-Sent Events on GET /sse.
    • Implements 11 tools, live resources (system://devices/status, system://logs/events), and prompt templates with OpenTelemetry tracing headers.
  2. AWS Bedrock Multimodal Vision Engine (AWS Builder Track):
    • Target Foundation Models: Anthropic Claude 3.5 Sonnet (anthropic.claude-3-5-sonnet-20241022-v2:0) and Amazon Nova.
    • Dual-Engine Architecture: Runs against live AWS Bedrock when credentials are configured, and seamlessly activates a high-fidelity smart simulation fallback when credentials are absent — ensuring hackathon evaluators encounter zero runtime failure.
    • Structural Motion Pre-Filter: Discards ambient noise (wind, headlights) before invoking multimodal models when pixel delta is below threshold: $$\Delta_{\text{motion}} < 20\% \implies \text{Filter Suppresses Token Spend}$$
  3. 50-Frame Multimodal Vision Benchmark:
    • Ground-truth dataset in data/benchmark-frames/ evaluated via npm run benchmark:
      • Overall Accuracy: 94.0% (47 / 50 frames)
      • Precision: 93.3% | Recall: 96.6%
      • F1 Score: 94.9%
      • Mean Inference Latency: ( \mu = 336\,\text{ms} )
      • Token Savings: ( \sim 11,200\,\text{tokens} ) saved via pre-filtering. $$ \begin{pmatrix} \text{TP} = 28 & \text{FP} = 2 \ \text{FN} = 1 & \text{TN} = 19 \end{pmatrix} \quad \implies \quad F_1 = 94.9\% $$
  4. Simulated Alexa+ Multimodal Living Room Web App:
    • Built with React 19, TypeScript, and Vite.
    • Features glowing animated Alexa Halo, browser Web Speech API voice synthesis and recognition, live Ring camera canvas, and 4K Fire TV player with animated PiP insets.
  5. Physical Hardware 1:1 Architecture:
    • Ring: Webhooks with HMAC-SHA256 signature verification (POST /api/ring/webhook).
    • Fire TV: Companion Silk Browser page (/firetv/display) and SSE overlay stream.
    • Bee Wearable: Standard Web Bluetooth API GATT services (Heart Rate 0x180D, Haptic 0xFFE1).

Challenges we ran into

  1. MCP Streamable HTTP Specification Transition: Migrating from legacy stdio MCP transports to the latest Streamable HTTP specification (Nov 2025) required implementing custom Server-Sent Events session tracking, defensive CORS whitelisting, and strict JSON-RPC 2.0 schema response compliance.
  2. Audio Collision & Acoustic Ducking: Early iterations experienced jarring overlap when Alexa spoke while Fire TV media played. We calibrated an audio ducking ramp that lowers media volume to ( 15\% ) within 200ms before spoken voice synthesis begins.
  3. Zero-Friction Evaluator Experience: We wanted hackathon judges on any OS (Windows, Mac, Linux) to test 100% of the cross-device experience in seconds without needing physical hardware or AWS billing setup. We designed a self-healing Pre-Flight Doctor (npm run doctor) and Dual-Engine AI runtime to achieve zero-friction onboarding.

Accomplishments that we're proud of

  • True Ecosystem Synergy: The first project that connects Alexa+, Ring, Fire TV, Bee, and AWS Bedrock into one harmonious ambient workflow.
  • 100% Verifiable Tests: 16 automated integration tests (npm run test:mcp) + 15 unit tests (npm run test:unit) + 6 frontend component tests passing with zero failures.
  • Judge Auto-Pilot (90s Guided Tour): Evaluators can press [A] on their keyboard to watch all 3 scenarios run hands-free with timestamped narration overlays.
  • Pre-Flight Doctor (npm run doctor): Instant verification of Node version, ports 3001 and 5173, dependencies, and dataset integrity before boot.
  • Comprehensive Documentation: Full friction logs (+10% bonus), developer product feedback, commercial enterprise B2B pricing model, and physical hardware setup guides.

What we learned

  • The Power of MCP: Rather than locking devices into proprietary skill silos, the Model Context Protocol allows AI agents like Alexa+ to dynamically discover, inspect, and invoke ambient tools across heterogeneous hardware.
  • Ambient Intelligence Over Notification Spam: Smart homes shouldn't just send loud notifications. By fusing vision reasoning with situational context (e.g., watching a movie vs sleeping), home automation becomes subtle, respectful, and genuinely helpful.

What's next for OmniLiving+: Autonomous Living Room Copilot for Alexa+

  • Official Vega OS Fire TV App: Porting the companion display into a native React Native / Vega OS background service.
  • Physical Bee Wearable Bluetooth Bridge: Publishing a Web Bluetooth daemon for zero-touch pairing with physical Bee smart rings and bands.
  • Multi-Tenant Senior Care Deployment: Deploying our HIPAA-compliant, zero-video-retention ambient fall detection system across commercial assisted living facilities ($35–$65 / resident / month).

Built With

  • alexa-plus
  • amazon-nova
  • aws-bedrock
  • bee-wearable
  • claude-3-5-sonnet
  • fire-tv
  • json-rpc-2.0
  • mcp
  • model-context-protocol
  • node.js
  • opensource
  • react
  • ring
  • sqlite
  • sse
  • streamable-http
  • tailwindcss
  • typescript
  • vite
  • web-speech-api
Share this project:

Updates

Submission history