Retina

Inspiration Retina was inspired by a simple problem: most AI assistants can answer questions, but they still expect users to see a screen and manually complete the task. For blind and low-vision users, describing the next step is not enough—the assistant should be able to perceive the environment, operate the computer, and finish the task. We designed Retina around one principle: the audio is the interface; the screen is telemetry. Every important action can be started, understood, confirmed, interrupted, and completed through voice. What it does Retina is a voice-first assistant for blind and low-vision users. A user can speak or type a request, and Retina routes it to the right specialized agent. Retina can:

Describe scenes and objects using a phone camera.
Scan pantry or refrigerator items and identify what is running low.
Navigate websites and complete browser-based tasks.
Control macOS applications, including Apple Notes.
Perform coding and file-management tasks within a restricted workspace.
Handle shopping workflows with confirmation steps and spending limits.
Remember conversations using Elasticsearch or local fallback storage.
Speak its progress, results, and errors instead of relying on visual indicators.
Stop its reasoning and audio immediately when interrupted. Retina is available through a macOS Electron overlay, a Next.js web client, and a VoiceOver-first SwiftUI Lens app. How we built it Retina uses a layered architecture connecting voice, vision, specialized agents, and computer control. The Electron and web clients capture microphone input and communicate with an Express and Socket.IO server. Gemini handles speech recognition, speech synthesis, and visual understanding. A LangGraph supervisor powered by Anthropic models classifies each request and routes it to a general, coding, commerce, desktop-control, or documentation agent. Compound requests can move through multiple agents. For example, a request to open an editor and write a function can first use the desktop agent and then the coding agent. We used:
Electron, React, and Vite for the macOS overlay.
Next.js for the browser client.
SwiftUI for the iOS Lens experience.
Express and Socket.IO for real-time communication.
LangGraph for routing and multi-agent coordination.
Gemini for speech, synthesized audio, and vision.
Anthropic models for classification and agent reasoning.
Stagehand for browser automation.
Native macOS accessibility controls for desktop automation.
Elasticsearch for long-term memory, with local-file fallback.
Three.js for the standalone audio visualizer. Challenges we ran into The biggest challenge was creating an interface that works without requiring the user to look at the screen. A loading spinner is meaningless to someone who cannot see it, so Retina must narrate long-running actions, failures, routing decisions, and completion states without becoming overwhelming. Speech recognition introduced another challenge, especially in noisy environments. We added typed input as a fallback and designed consequential actions—such as purchases—to repeat what Retina heard before continuing. Interruptibility was also difficult. Stopping visible output was not enough; we needed to cancel active reasoning, pending tools, generated speech, and audio playback together. Desktop and browser automation introduced reliability and permission issues. Retina has to manage macOS Accessibility permissions, changing interfaces, network failures, and unavailable API services. We built health checks, graceful fallbacks, and a preflight system to catch problems before a session begins. Finally, coordinating several agents while keeping conversation state consistent required careful routing, tool boundaries, and status reporting. Accomplishments that we're proud of We are proud that Retina closes the loop from perception to action. It can identify what is in front of a user, understand what needs to happen, and complete the task on the user’s device. Other accomplishments include:
Building a working supervisor that routes requests among five specialist agents.
Supporting multi-agent workflows for compound requests.
Creating a self-voicing interface that does not depend entirely on a screen reader.
Making every active operation interruptible.
Adding confirmation gates and spending limits to commerce workflows.
Connecting vision, pantry scanning, and reordering in one experience.
Supporting desktop, browser, web, and mobile interfaces through one backend.
Providing both cloud-backed memory and an automatic local fallback.
Making agent activity observable without turning the product into a technical dashboard. What we learned We learned that accessibility cannot be added at the end of development. It changes the architecture of the entire product. A system is not truly voice-first simply because it accepts speech. It must also communicate state, recover from misunderstandings, support interruption, and avoid silent waiting. Confirmation is especially important when speech recognition is connected to consequential actions. We also learned that specialized agents are easier to control and evaluate than one agent with unlimited access. Clear tool boundaries, restricted workspaces, purchase limits, and explicit confirmation steps make the system more predictable and safer. Finally, we learned the importance of graceful degradation. Voice input can fail, APIs can become unavailable, and permissions can change. An assistive product must explain these failures and provide another path forward. What's next for Retina The next step is testing Retina with blind and low-vision users and letting their feedback guide the interaction model. We also plan to:
Improve speech recognition in noisy environments.
Reduce the latency between speaking and receiving a response.
Add more offline speech and vision capabilities.
Expand desktop support beyond macOS.
Support more productivity, communication, and shopping applications.
Improve personalization while keeping user data private.
Add stronger verification and recovery for consequential actions.
Expand environmental understanding beyond individual images.
Make compound agent workflows more reliable and transparent.
Package Retina as a polished application that can be installed without developer setup.