REACH

Tell Alexa what you want. Your devices make it happen.

Comment

Inspiration

Like almost every developer growing up on science fiction, I wanted my own Jarvis. When smart speakers like the Amazon Echo Dot first entered our living rooms and desks, they promised that ambient computing future. Yet in practice, voice assistants hit a hard wall the moment a task leaves their walled garden: they can dim a smart bulb, play a podcast, or check the weather, but they cannot open an assignment PDF on your MacBook, transcribe a late-night sync recording sitting in your Downloads folder, or sync research notes directly to your phone.

The gap wasn't intelligence—modern foundation models possess immense reasoning power. The missing piece was reach. I wanted my Echo Dot to serve as the ambient voice gateway to my actual compute environment—orchestrating my laptop, browser, and mobile device into a unified, actionable mesh. That vision became REACH.

What it does

REACH transforms natural voice input into cross-device execution through the Model Context Protocol (MCP) and Amazon Bedrock:

  • Voice-First Goal Routing: Speak naturally to an ambient interface or Echo Dot ("Reach, open the assignment PDF on my desktop, solve problem 3, and launch the Word document").
  • Autonomous Planner & Execution Mesh: Decomposes complex spoken goals into discrete primitives across connected devices, utilizing AWS Bedrock (Claude 3.5 Sonnet) as the central reasoning engine.
  • Hero Workflow: Ingests meeting audio on a laptop, stages it securely via Amazon S3, transcribes multi-speaker dialogues with Amazon Transcribe, synthesizes executive meeting minutes via Bedrock, and formats and launches a native .docx document locally.
  • Consequential Policy Engine: Routine informational actions execute instantly, while irreversible or sensitive actions (such as uploading audio recordings or deleting files) pause for a dedicated visual and voice approval gate before execution proceeds.
  • Cross-Device Clipboard Sync: Encrypts and transfers data directly across nodes (e.g. laptop research summary to mobile phone clipboard).
  • Low-Latency Spoken Synthesis: Streams audio replies back to the user using neural AI speech synthesis with expressive emotional prosody.

How we built it

REACH is architected around a stateless, protocol-first design pattern connecting cloud reasoning with local device capabilities:

[ Ambient Voice / Echo Dot ]
            │
            ▼
[ Express MCP Server (:3000) ] <---> [ AWS Bedrock (Claude 3.5 Sonnet) ]
            │                                  │
            ├── [ Policy Gate Engine ]         ├── [ Amazon S3 ]
            │                                  └── [ Amazon Transcribe ]
            ▼
[ Local Device Mesh / Node Daemon (:3001) ]
   ├── Local Filesystem & Office Automation (.docx generation)
   ├── Playwright Headless/Headed Browser Engine
   └── Cross-Device E2E Encrypted Clipboard

### Technical Formulation & Planning Model

We modeled device orchestration as a dynamic capability graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, where vertices $\mathcal{V}$ represent active registered devices and edges $\mathcal{E}$ define allowable cross-device data transports (e.g., laptop-to-cloud audio staging, laptop-to-phone clipboard synchronization).

For any high-level user prompt $u$, the planner synthesizes an ordered execution sequence of tool invocations $A = \{a_1, a_2, \dots, a_n\}$ subject to our safety policy mapping function $\Phi$:

$$\Phi(a_k \mid s_k) \in \{\text{Allow}, \text{Gate}, \text{Deny}\}$$

Where state $s_k$ tracks action scope, file path blast radius, and network egress boundaries:

$$\text{Decision}(a_k) = \begin{cases} 
\text{Execute}(a_k) & \text{if } \Phi(a_k \mid s_k) = \text{Allow} \\ 
\text{RequestApproval}(a_k, \tau) & \text{if } \Phi(a_k \mid s_k) = \text{Gate} \\ 
\text{Halt}(\text{Reason}) & \text{if } \Phi(a_k \mid s_k) = \text{Deny} 
\end{cases}$$

Here, $\tau$ represents the temporal countdown window before an unconfirmed consequential action automatically expires and aborts.

### Core Stack:
* **Reasoning & Planning**: AWS Bedrock (`anthropic.claude-3-5-sonnet-20241022-v2:0`)
* **Audio & Storage Infrastructure**: Amazon Transcribe, Amazon S3
* **Protocol & Connectivity**: Model Context Protocol (MCP 2025-11-25 Streamable HTTP)
* **Local Automation**: Node.js, TypeScript, Playwright, native `.docx` generator
* **Speech Synthesis**: Edge Neural AI TTS (`en-US-JennyNeural`) streaming 24kHz audio
* **Motion Graphics & Presentation**: Remotion programmatic video framework

## Challenges we ran into

1. **Bridging Cloud Reasoning with Local Isolation**: Cloud-hosted LLMs cannot reach into local home networks without exposing open inbound ports or static public IPs. We solved this by implementing an MCP agent daemon that registers outward-facing capability heartbeats to the central server, pulling tasks securely without exposing local attack surfaces.
2. **Preventing LLM Path Hallucination**: Early iterations saw the planner invent paths like `~/Documents/assignment.pdf` when files lived in arbitrary project subfolders. We introduced a mandatory two-phase primitive pattern: deterministic fuzzy file indexing (`files.search`) followed by explicit ground-truth path citation before invoking downstream transformers.
3. **Stream Synchronization & Asynchronous Voice Autoplay**: Modern browser security policies strictly block programmatic audio autoplay unless bound to direct user activation. Implementing low-latency neural TTS streaming required unlocking a persistent HTML5 audio context on initial pointer interaction and handling duplex fallback to system speech synthesis with zero audible latency spikes.

## Accomplishments that we're proud of

* **Zero Mocks End-to-End**: Every single demo action executes on real local machine state—actual files are found, actual Word documents are generated on disk, actual browsers are launched via Playwright, and real audio files are uploaded and transcribed.
* **Sub-30-Second Hero Workflow**: A full 105-second four-voice product meeting recording is searched on disk, uploaded to S3, transcribed via Amazon Transcribe, synthesized into actionable minutes by Claude 3.5 Sonnet on Bedrock, and launched on the desktop in $\approx 22$ seconds.
* **Strict Consequential Safety Boundaries**: Built a real human-in-the-loop security architecture into an ambient voice flow, proving that voice interfaces do not have to compromise on security to feel magical.

## What we learned

* **Protocol-Driven Extensibility**: Building against the Model Context Protocol proved that standardization beats custom glue code. Adding a new operating system capability (like Playwright browser automation or document generation) required only registering a tool schema rather than rebuilding the core prompt loops.
* **The Balance of Autonomy vs. Oversight**: Complete agentic autonomy is dangerous without clear security boundaries. Dividing primitives into read-only actions and policy-gated operations creates an assistant that feels fast and frictionless without ever running dangerous operations behind your back.

## What's next for REACH

* **Native Echo Skill Integration**: Packaging the client directly as an Alexa+ custom skill once developer APIs open broadly.
* **Multi-Node Swarm Execution**: Distributing heavy computational tasks across multiple specialized machines (e.g., delegating local model fine-tuning to a GPU workstation while running browser steps on a laptop).
* **Zero-Network Local Fallbacks**: Adding on-device small language model (SLM) routing so fundamental device commands remain responsive even when internet connectivity drops.

Built With

Share this project:

Updates

Submission history