Inspiration
In 2023, distracted driving killed 3,275 people and injured about 324,819 in the U.S., according to NHTSA (NHTSA). Worldwide, drivers using a phone are about 4 times more likely to crash (WHO, via GRSP). Telling people “don’t text and drive” hasn’t fixed it, because people still need to reply, navigate and change the song. Siri and CarPlay only work in a few apps. I wanted to remove the reason to pick up the phone at all.
What it does
Relay is a voice agent that uses your phone for you, so your eyes stay on the road. Tap a Shortcut on your iPhone (or Apple Watch) and say “Text Mom I’m running late,” “Navigate to the nearest Starbucks,” “Next song,” or “Morning outlook.” Relay opens the right app, reads the screen and taps through it like a person would, then answers out loud.
Works in any app: Messages, Maps, Gmail, Instagram, LinkedIn, YouTube Music, X. It isn’t limited to the apps CarPlay supports. Never makes you look: it asks “Should I send it?” out loud and listens for “yes.” After each task it asks “Anything else?” Say “No, I’m done” to finish. Safe by default: it confirms before sending, paying, following or connecting, never types passwords or card numbers, and stops on payment or verification screens. Also for blind users: the same no-screen design helps anyone who can’t use a screen, not just drivers.
How we built it
Brain: a Python FastAPI server deployed on a Vultr cloud server with HTTPS. A helper on my Mac connects over a websocket and controls an Android phone (Pixel emulator) through uiautomator2/ADB, or the Mac itself. Three AI models: Jev (via OpenRouter) makes fast tap-by-tap decisions, Claude plans tasks and handles hard steps, and Gemini describes the screen. Voice: an iOS Shortcut sends what you say to the server and loops until you’re done. Replies come back in an ElevenLabs voice, and the server can transcribe recorded audio itself. Instant skills handle music, navigation and opening apps with no AI delay. A live dashboard shows every step, and 463 automated tests run on GitHub Actions. Challenges we ran into Siri and the Watch: Siri kills long Shortcuts. I rebuilt the loop so the server answers every few seconds, and you can answer “Should I send it?” even after Siri quits. Agents getting stuck: in Maps it looped on popups and old routes. I added loop detection, permission handling, location syncing and a hard time limit. Apps blocking emulators: Spotify and X refused emulator logins, so I switched to YouTube Music and added an X-in-browser mode. Tiny bugs, big effects: one null value silently crashed the iPhone Shortcut. The live dashboard is how I found it.
Accomplishments that we’re proud of
It sends real texts, navigates and summarizes email and news, by voice, end to end. It’s deployed in the cloud, not just running on a laptop. Safety confirmations are built into the core. 463 tests, and a dashboard that makes every AI decision visible.
What we learned
A hands-free product has to make sense with zero glances: every reply, question and error has to work out loud. Mixing a fast model with a strong one beats using either alone. The hardest problems were the edges: timeouts, popups, permissions and platform limits.
What’s next for Relay
A native Apple Watch app and CarPlay support, so it starts with one button on the steering wheel. Running on real phones, not just the emulator. An auto-reply when you’re driving (“I’m driving, Relay will read me your message”), plus user-set rules. Testing with drivers, blind users and road-safety groups.
