BeaconOS

Voice-first spatial OS uniting Alexa+ MCP, Fire TV, and Ring into an ambient living room command bridge for accessibility, hands-free safety, and AI vision.

Comment

BeaconOS — Ambient Voice OS and Spatial Living Room Bridge

A cross-platform, AI-driven operating architecture uniting Fire TV, Alexa+ MCP, Ring, and mobile devices into an accessible, distraction-free ambient ecosystem.

Inspiration

I started this project with a focused architectural goal: I wanted to build an AI-first, cross-platform system to organize my time, schedule deep work, and manage daily priorities without being held hostage by traditional, distraction-heavy smartphone interfaces. The objective was an eyes-free, voice-first environment that would keep technology in the background and eliminate cognitive clutter.

While engineering this screen-agnostic interaction model, I had a profound realization: an operating architecture built to function seamlessly without looking at a screen is, by its very nature, the exact tool a visually impaired or blind person needs every day. If an eyes-free, voice-driven system gives a sighted person their focus and time back, for someone who cannot see, it gives them their independence, spatial awareness, and safety back.

That realization expanded the project from a personal productivity engine into an ambient living room ecosystem. A smartphone alone is insufficient when misplaced or out of reach. By integrating the smart devices already present in our homes—Amazon Echo speakers via the Model Context Protocol (MCP), Fire TV displays, and Ring perimeter cameras—the living room itself becomes an ambient physical bridge, ensuring that assistance, navigation, and emergency safety are always accessible hands-free.

What It Does

BeaconOS connects three everyday devices into one calm, unified operating system: the phone in your pocket, the Fire TV in your living room, and the Ring camera at your front door.

Instead of searching through confusing grids of app icons, navigation works like walking through rooms in a house. With simple two-finger swipes and spatial audio sounds, users can move naturally between their daily cockpit, focus timer, emergency radar, and AI camera without needing to stare at a screen.

1. Fire TV: Big-Screen Comfort

Many visually impaired individuals struggle with small phone screens. We brought BeaconOS to Fire TV because large living room displays make reading comfortable and reduce severe eye strain.

  • Today: Users can sit on their couch and manage their daily priorities, alarms, and emergency radar using only the TV remote, with complete privacy and zero logins.
  • In the Future: We plan to expand this big-screen comfort to accessible book reading and voice-assisted web browsing.

2. Alexa+ and MCP: Hands-Free Room Bridge

Using a self-hosted Model Context Protocol (MCP) server running on the modern Streamable HTTP standard, any Amazon Echo device can control the phone across the room.

  • Finding a Lost Device: If a blind user misplaces their phone, they simply ask Alexa. The system immediately rings a loud acoustic siren, flashes the camera light, and vibrates the phone, while displaying an active visual radar card on the Echo Show screen.
  • Remote Elevator: Users can tell Alexa to move the phone's screen between floors (Cockpit, Settings, or Vault) completely hands-free.

3. Ring Cameras: Private Home Perimeter

  • Today: BeaconOS integrates Ring cameras to give users immediate descriptions of who or what is outside when the doorbell rings, alongside two-way voice talk.
  • In the Future: We are designing an on-device recognition model that identifies familiar family members or friends and announces them by name, keeping all personal recognition data strictly private inside the home.

4. An Open Ecosystem Built with the Community

BeaconOS is not a closed app; it is an open-source platform designed to grow. Blind users can share their daily challenges, and developers can build and plug in new spatial rooms in minutes. AI acts as the central brain handling voice understanding, camera vision, and hardware actions. It runs on Android and Windows today, with an architecture ready to expand to iPhone.

5. Fair Access and Safety

Every device creates its own private, self-custodial identity. Through transparent smart contracts on Arbitrum Sepolia, community donations subsidize the cost of AI vision scans for blind users and secure emergency SOS alerts so they can never be altered or lost.

How We Built It

BeaconOS is designed as a modular, local-first system that connects multiple devices without complicated setups:

1. Unified Multi-Platform App (Flutter)

We built the core application using Flutter and Dart, allowing a single codebase to run across Android phones, Fire TV, and Windows desktop, with an architecture ready for iOS. Instead of hardcoded features, we built a modular room system: developers can create a new spatial room simply by defining its coordinates and voice actions. Navigation uses smooth mathematical spring physics, so moving between rooms feels tactile and fluid.

2. Alexa+ MCP Server (Python and FastAPI)

To bridge Alexa with physical devices, we built a self-hosted Model Context Protocol (MCP) server using Python and FastAPI. It follows the modern 2025-11-25 Streamable HTTP specification using a single unified endpoint (/mcp). When an Echo speaker receives a command, the server executes the tool, returns multimodal visual cards for Echo Show smart displays, and immediately signals the phone to activate its siren and strobe light. We also developed a zero-install web simulator to test voice queries and inspect JSON-RPC traffic.

3. Local-First Vault and Fast Wi-Fi Sync

Every piece of data is stored locally in an encrypted SQLite database using Drift. The system works completely offline. Within the home, devices find each other silently using local UDP discovery on port 8491 and exchange messages over direct WebSockets on port 8492. This local connection transfers actions in less than 2 milliseconds without touching the internet, while falling back to Supabase cloud relay only when away from home.

4. AI and Visual Understanding

We integrated multimodal AI to act as the digital eyes of the system. Camera snapshots and Ring doorbell events are analyzed to generate clear, concise descriptions of surroundings, read printed documents, or identify visitors. We also integrated the Amazon Bedrock Converse API, creating an automatic translation layer that converts standard MCP tool schemas into Amazon Bedrock tool specifications.

5. Sovereign Identity and Safety

There are no passwords or account registrations. Each installation generates its own cryptographic key directly on the hardware. This identity connects to the BeaconImpactPool smart contract on Arbitrum Sepolia, where community donations subsidize AI costs for blind users and record emergency SOS events as a permanent, tamper-proof audit trail.

Challenges We Ran Into

Building BeaconOS presented several real architectural and design challenges:

1. Mobile Spatial Gestures vs. Content Scrolling

Designing the movement system on mobile was surprisingly difficult. Initially, I wanted users to swipe with a single finger to slide between rooms. However, this immediately conflicted with the lists, tasks, and cards inside each room. The system could not tell if the user wanted to scroll down a list or move to the next room. To solve this, I made a clear design decision: use simple two-finger gestures to travel between spatial rooms, leaving single-finger touches for interacting with content inside the room.

2. The Web 2.5 Learning Curve

The concept of Web 2.5 and decentralized identities (DIDs) was completely new to me. I had to spend significant time studying how self-custodial keys and smart contracts on Arbitrum function before writing the integration code. It took effort to understand, but it proved to be the right choice because it gave blind users a secure system with zero passwords and zero account registrations.

3. Engineering an Extensible System Takes Time

Designing BeaconOS as an open, modular ecosystem rather than a rigid app consumed a tremendous amount of development time. Creating a pluggable room architecture where any developer can register new coordinates and voice actions was challenging. It is not 100% perfect yet, but I believe it was the right decision because the core foundation is now ready to scale and grow organically.

4. Cross-Platform Realities

Developing and testing iOS builds directly from a Windows workstation is impractical and adds friction. Instead of splitting my focus and slowing down momentum, I made the pragmatic choice to prioritize Android, Fire TV, and Windows desktop first, while keeping the underlying Dart codebase clean and structured so it can expand to iPhone later.

5. Alexa+ and the Early Nature of MCP

The Model Context Protocol (MCP) and Alexa+ integration are still relatively new. While we established a working Streamable HTTP server for essential actions like pinging misplaced hardware and controlling the phone's elevator, the ecosystem is still maturing. I have not yet integrated the full depth of advanced features to the level that completely satisfies my personal standards, which leaves exciting room for future development.

Accomplishments That We're Proud Of

  • Building a Screen-Agnostic System: We created an operating environment that can be fully navigated without looking at a screen, using natural spatial audio and simple two-finger gestures.
  • A Working Living Room Bridge: We successfully connected an Alexa+ Streamable HTTP MCP server, a Fire TV Leanback dashboard, and an Android phone into a unified experience with real-time response times.
  • True Offline-First Independence: Every task, alarm, contact, and setting is saved locally in an encrypted database, ensuring the system remains completely functional even without internet connectivity.
  • Open-Source and Community-Ready: We released the entire platform as open-source code under the MIT license, establishing an extensible foundation that developers anywhere can build upon.

What We Learned

  • Accessibility Means Rethinking Interaction: Real accessibility is not just adding large text or voice synthesis on top of existing apps. It requires redesigning interaction from the ground up so that spatial awareness and sound replace visual clutter.
  • The Physical Power of MCP: The Model Context Protocol (MCP) is more than just a way to build chatbots. It is an open standard that allows ambient living room assistants to trigger immediate, physical actions on remote hardware.
  • Pragmatic Engineering Choices: We learned the importance of practical focus—such as choosing simple two-finger navigation to resolve screen-scrolling conflicts, and concentrating on Android and Windows to ensure rock-solid stability before expanding to other platforms.

What's Next for BeaconOS

  • Big-Screen Reading on Fire TV: We want to expand the Fire TV experience into comfortable, high-contrast book reading, document review, and voice-assisted web browsing to eliminate eye strain for low-vision users.
  • Private Local Visitor Recognition: We plan to develop lightweight, on-device models for Ring cameras that recognize family and friends and announce them by name, keeping all personal recognition data private within the home.
  • Official iOS Expansion: We will bring the BeaconOS Flutter engine to iPhone once dedicated macOS build pipelines are integrated.
  • Community-Driven Spatial Rooms: We will provide open documentation and starter templates so blind users can propose everyday workflows, and developers around the world can build new spatial rooms to solve them.

Built With

Share this project:

Updates

Submission history