Inspiration

Over 70 million individuals worldwide live with speech impairments, non-verbal conditions, or are seniors living independently with high fall risks. Traditional smart assistants make a single, dangerous assumption: they assume the user can speak loudly, clearly, and hear verbal replies.

When an elderly resident suffers a ground-level fall, an acute stroke, or an intruder situation, shouting "Alexa, call 911" is impossible. We asked ourselves: What if Alexa+ could see, understand silent gestures, and act as an ambient guardian without requiring a single spoken word?

That is why we built Aura+: the world’s first Multimodal Silent Guardian and Ambient Accessibility Agent engineered for next-generation Alexa+.

What it does

Aura+ transforms Alexa+ from a passive, reactive smart speaker into a proactive, multimodal ambient intelligence:

Silent & Gestural Interaction: Uses real-time computer vision to detect non-verbal American Sign Language (ASL) distress signs (such as Open Palm, Chest Pain indicators, and Thumbs Up) along with ground-level fall posture anomalies.
Self-Hosted Streamable HTTP MCP Server: Implements the latest Model Context Protocol (spec 2025-11-25+) with Server-Sent Events (SSE) streaming progress chunks across 5 specialized tools: mcp_triage_multimodal, mcp_ring_telepresence, mcp_lifeline_dispatch, mcp_generative_ui_card, and mcp_accessibility_translator.
AWS Bedrock Multimodal Reasoning: Connects to Anthropic Claude 3.5 Sonnet on AWS Bedrock to evaluate visual context, eliminate false alarms, and synthesize clinical triage summaries.
Ring Smart Home Telepresence: Inspects perimeter cameras, triggers automated outdoor intercom guidance, and automatically unlocks the Ring Smart Deadbolt so first responders never have to break down a door.
Tactile Generative UI (A2UI): Renders dynamic, high-contrast decision cards with large touch targets and auto-dismiss countdown timers for low-dexterity or non-verbal users.

How we built it

Alexa+ Primary Track: Built a self-hosted Streamable HTTP MCP Server in Node.js/TypeScript conforming to the November 2025 MCP specification, handling JSON-RPC 2.0 requests and chunked SSE streaming over /mcp/stream.
AWS Builder Mini Challenge: Integrated @aws-sdk/client-bedrock-runtime using the Bedrock ConverseStream API with Claude 3.5 Sonnet for multi-modal reasoning and clinical dispatch synthesis.
Frontend & 10-Foot UI: Developed a multi-view application in React 18, Vite, and Tailwind CSS adhering to an accessible color palette: Charcoal Grey (#2A2A2A), Moonlit Silver (#C0C0C0), and Signal Red (#E10600).
Ring Device Simulator: Built a virtual device simulator emulating live camera feeds, motion webhooks, doorbell chimes, and smart deadbolt locks.
Ambient Voice & Accessibility: Integrated the browser Web Speech API for natural text-to-speech reassurance and live audio waveforms.

Challenges we ran into

Browser-Side MCP Streaming: Native browser EventSource only supports HTTP GET. We implemented a custom stream consumer using fetch() and ReadableStreamDefaultReader with UTF-8 chunk decoding to parse JSON-RPC Server-Sent Events over HTTP POST.
Bedrock Multi-Modal Binary Inputs: AWS Bedrock Converse requires raw Uint8Array binary byte buffers rather than standard base64 data URIs. We designed an automated binary parser and sanitizer pipeline.
Real-Time Hand Tracking Performance: Ensuring skeletal landmark tracking and bounding boxes rendered at a smooth 60 FPS while background workers streamed video frames to Bedrock.

Accomplishments that we're proud of

Successfully deploying a production-grade, spec-compliant Streamable HTTP MCP Server (2025-11-25+) executing 5 custom tools live.
Creating an emotionally impactful, life-saving accessibility product with a complete 8-screen dashboard that anyone can test with a webcam or simulated feeds.
Compiling an extensive, actionable Developer Friction Log (FRICTION_LOG.md) to assist Amazon engineering teams with MCP and Bedrock developer experience, qualifying for the +10% judging bonus.

What we learned

The Model Context Protocol (MCP) represents the future of agentic AI, turning smart assistants from closed voice silos into modular, streamable tool orchestrators.
Cross-device synergy—correlating visual context from room cameras with exterior Ring perimeter sensors—virtually eliminates false alarms in domestic elder care.

What's next for AURA+

Fire OS & Vega OS Native Port: Packaging Aura+ for Echo Show 15 and Fire TV living room deployment.
Edge ML Acceleration: Running on-device MediaPipe models directly on local NPU silicon for zero-latency offline detection.
911 CAD Integration: Direct automated webhook integration with Computer Aided Dispatch systems for instantaneous HL7 emergency telemetry transfer.