Inspiration
What it does
Inspiration and Background
When an elderly patient living independently experiences acute physiological distress—such as heart attack symptoms, breathing difficulty, or a fall—the window for intervention is narrow. The standard emergency response sequence breaks down in two critical places:
- Patient Incapacitation: The individual is frequently unable to locate a phone, navigate a touch screen, or clearly communicate their location and medical history during severe pain or distress.
- First Responder Access Delay: Paramedics typically lose 2 to 6 minutes attempting to breach locked doors or waiting for access codes, resulting in critical treatment delays.
Current smart home voice assistants remain passive: they wait for explicit voice commands and return basic web answers. They do not connect clinical triage logic with physical home access control.
With Amazon opening developer integration for Alexa+ and the Model Context Protocol (MCP), we built a voice-first agent capable of evaluating clinical symptom severity and executing real-world physical actions—specifically, opening Ring smart locks for dispatched emergency crews and coordinating family care circles.
What It Does
CareSync operates as an ambient service on an Echo Show 15 smart display, combining structured tool calling via MCP with Amazon Bedrock:
Clinical Symptom Triage: Ingests spoken symptom descriptions alongside duration and medical history. Calculates clinical risk scores (Critical, Moderate, Mild) to determine immediate action.
Autonomous Emergency Home Access: When critical severity is detected, CareSync does not stop at voice confirmation. It automatically unlocks Ring smart door locks so first responders can enter immediately, turns on pathway lights, and sends urgent SMS alerts with vitals to family caregivers.
Medication Tracking and Allergy Detection: Manages daily prescriptions with time-of-day intake status. Checks against documented patient drug allergies (such as Penicillin) prior to confirming intake.
Continuous Vitals Monitoring: Aggregates continuous telemetry from connected wearables and sensors: resting heart rate, blood pressure, blood oxygen (SpO2), and blood glucose, rendering live trendlines on the display.
Telehealth Scheduling: For moderate or non-emergent symptoms, queries provider availability through Amazon Clinic protocols and commits calendar events to the Echo Show schedule.
Developer Protocol Console: Includes a built-in protocol drawer displaying raw JSON-RPC 2.0 frames, real-time event stream heartbeats, token usage, and execution latency.
How We Built It
CareSync is built on a full-stack agentic architecture:
- Model Context Protocol (MCP) Server: Built with Node.js and Express implementing the official Model Context Protocol 2024-11-05 specification over Streamable HTTP (POST /mcp) and Server-Sent Events (GET /sse). It exposes 6 tools, 3 structured healthcare resources, and 2 prompt templates.
- Amazon Bedrock Multi-Agent Brain: Powered by Anthropic Claude 3.5 Sonnet on Amazon Bedrock using the Converse API. When AWS credentials are configured, it runs live multi-step tool calls. For frictionless judging, it includes a zero-config fallback engine that replicates identical tool chains without requiring API keys.
- Echo Show 15 Multimodal Interface: Built with React and Tailwind CSS. Features an authentic Amazon hardware aesthetic with the signature cyan ambient light bar at the bottom edge that reacts in real-time to voice states (Listening, Processing, Speaking, and Red Emergency Strobe).
Challenges We Overcame
Schema Mapping Between MCP and AWS Bedrock: The Model Context Protocol specifies tool arguments as standard top-level JSON Schema, whereas the Amazon Bedrock Converse API requires an explicit nested structure. We built an automated bidirectional schema adapter in our backend that maps MCP declarations into Bedrock-compliant specifications with strict validation.
Fast Emergency Dispatch Under Multi-Step Reasoning: Chaining multiple AI reasoning passes introduces noticeable latency, which is unacceptable in cardiac emergencies. We structured clinical triage prompt instructions to permit atomic multi-tool dispatch in a single inference turn when acute keywords are recognized, reducing dispatch latency to under 180ms.
Cross-Device Hardware Synchronization: Coordinating physical lock states, lighting levels, and UI updates across asynchronous HTTP streams caused race conditions during state resets. We centralized device state management within an in-memory transactional store, broadcasting real-time Server-Sent Events on every tool execution to keep the UI in sync.
Accomplishments
- Full MCP Standard Compliance: A working reference implementation of a self-hosted MCP server ready to pair with any MCP client or future Alexa+ environments.
- Real Physical Home Action: Taking real-world action during a crisis—unlocking doors for paramedics and activating emergency pathway lights during a distress call.
- Frictionless Testing: Judges and developers can clone the repository and run the entire stack with a single command (npm run dev) without needing AWS account setup.
What We Learned
- How the Model Context Protocol cleanly separates application tools (clinical triage, door locks) from the underlying LLM.
- Best practices for designing 10-foot ambient voice and visual user experiences for devices like the Amazon Echo Show 15.
- How to structure tool schemas in the Amazon Bedrock Converse API for reliable, zero-error tool selection.
What's Next for CareSync
- Respiratory Acoustic Biomarkers: Adding audio stream processing via wearable devices to passively detect nocturnal coughing or wheezing.
- Fire TV Ambient Standby: Bringing the CareSync dashboard to living room Fire TV screens during standby mode.
- EHR Integration: Connecting health resources directly to hospital FHIR endpoints for real-time prescription synchronization. ## How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for CareSync: Ambient Health & Home Access Agent
Built With
- amazon-bedrock
- claude-3.5-sonnet
- express.js
- healthcare
- iot
- mcp
- model-context-protocol
- node.js
- react
- ring
- speech-synthesis
- tailwindcss
- vite