Inspiration

Every one of us has a grandparent who has been quietly locked out of daily life.

Not out of technology — out of apps. Paying the electric bill now means an account and a password. Getting dinner means DoorDash. Getting to a doctor's appointment means Uber. Each one is a smartphone, an app store, a login, a two-factor code, and a UI redesigned every six months. The generation that needs these services most is the one least able to reach them.

And the moment they do go online, they become a target. Scam texts and fake-invoice emails are engineered specifically for them, and the financial and emotional damage is brutal.

So we stopped trying to teach older adults to use apps, and asked the opposite question: what if the app came to them, over the one interface they have used their entire life?

A phone call.

Gary is a phone number. Your grandmother calls it and says what she needs, in whatever words she'd use with a person. Gary pays the bill, orders the food, books the plumber, calls her with her medication reminder — and texts her daughter to approve anything risky.

No app. No login. No learning curve. It works on a landline.

What it does

You call an ordinary phone number and talk normally.

Bills. Gary reads the user's Gmail, finds bills, calls to remind them, and pays after reading the amount and the payee back out loud.
Food, rides, and home services. "Order me a large wings." "My sink is leaking." Gary finds the options, reads back the price, and places it.
Scam Guard. Every payment runs through a policy check. New recipients, amounts over the limit, suspicious payee names, and classic scam phrasing get paused, and a family member is texted to approve. The caller is never made to feel foolish — Gary just says it would like to check with their daughter first.
Daily reminder calls. Family schedules them from a web dashboard. If nobody picks up, Gary retries in fifteen minutes, then alerts the family.

Family sees every action on a dashboard and approves or denies by replying YES or NO to a text.

How we built it

The call. Twilio provides the phone number and streams raw call audio over a WebSocket. Our FastAPI backend bridges that stream to Deepgram's Voice Agent API, which handles speech-to-text, the conversation model, and text-to-speech in one socket. Turn-taking is deliberately tuned slow — a 15-second end-of-turn timeout — so Gary never cuts off someone who pauses mid-sentence to think.

The capabilities. Everything Gary can do is an MCP tool. Four FastMCP servers — check-ins, money, orders, mobility — run inside the same FastAPI app. Deepgram doesn't speak MCP natively, so we wrote an adapter that lists tools from all four servers when a call starts, converts them into Deepgram function definitions, and routes function calls back to the right server. The payoff: adding a new capability is adding one Python function with a docstring. The voice layer never changes.

The web that has no API. Food, rides, and home services don't expose APIs. So browser_agent/ drives real websites the way a person would — a real Chromium window, clicking real buttons on REAL's DashDish and Udriver clones and on TaskHare, a household-services site we built ourselves. Critically, it reads the page's accessibility tree rather than screenshots: it's what a screen reader sees, it's an order of magnitude cheaper than vision, and it turned out to be ~25% faster.

State and safety. Supabase holds fifteen tables — bills, orders, bookings, approvals, pending actions, reminders, and an activity log the dashboard renders. Gmail and Calendar are read through the real Google APIs and classified into bills, appointments, and suspected scams.

Challenges we ran into

A phone call can't tolerate 30 seconds of silence — but a browser order takes 30 seconds

This was the hard one, and it shaped the architecture.

Our first working order took 49 to 79 seconds. That's fine for a background job and completely unusable in a conversation. You cannot leave an 80-year-old holding a silent phone.

The fix was to stop treating the browser as something that happens after the conversation, and start running it ahead of the conversation. The moment the caller names a restaurant, a lookup job launches and parks the browser on the store page. When they name a dish, a second job carts it, hits checkout, and stops there — capturing the real price so the safety check runs on the actual number, not a guess. By the time the caller says "yes," there is exactly one button left to click.

That last step takes about eight seconds. We also added learned-flow replay: a repeated order ("my usual") replays a recorded flow in ~15 seconds with zero model calls.

We couldn't send a single text message

US carriers blocked SMS from our Twilio number — error 30034, A2P 10DLC registration, which takes days we did not have.

Rather than fake the approval flow for the demo, we put family messaging behind a channel switch and ran it over Twilio's WhatsApp sandbox instead. One environment variable moves it back to SMS the day registration clears. Nothing else changes.

Keeping the language model out of the safety path

The obvious design is to expose the safety check as a tool. It's also wrong — because then the model can simply not call it, and "we asked the model nicely not to wire money" is not a security model.

Instead:

policy.check() is called inside every money-moving tool. There is no code path to a payment that skips it.
Every action is split into prepare_* and confirm_*. prepare_* returns details to read aloud and mints a single-use pending action; confirm_* executes only against a valid, unexpired, unused pending-action ID belonging to the same user and action type.

No sequence of model outputs can move money without a human hearing the amount first.

A race condition with real consequences

If a family member approves mid-call, Gary must never say "I paid it" before the payment actually happened. We reordered the approval webhook so it runs the executor first, records the outcome, and only then injects the result into the live call. The caller only ever hears about things that are already true.

The model itself was a bug

gpt-4o-mini hallucinated tool results — cheerfully announcing it had found restaurants it had never looked up. Moving the conversation model to gpt-4.1 fixed it, and we added an explicit rule that Gary may never state an outcome before a tool has returned one.

What we learned

Voice is a latency problem wearing an AI costume. We expected to spend the weekend on prompting and model quality. We spent it on dead air. Almost every meaningful improvement came from making the system speculate ahead of the user rather than react to them.

Measure your own architecture instead of arguing about it. When we had two candidate decision engines for the browser agent, we didn't debate — we benchmarked, three runs each, and committed the results to the repo. A first-time order went from a 34.5s median to 27.1s. Interestingly the faster one takes more steps (11 vs 5) but decides in ~0.4s each, so per-step cost is browser time, not model time. We'd never have guessed that.

Designing for older adults is mostly subtraction. No app. No login. No menus. Never more than three options in a single turn. Never make someone feel stupid for almost falling for a scam. Every feature we cut made the product better.

What's next

Finishing the rides vertical, completing carrier registration so family messaging runs on plain SMS, live Google OAuth against a real inbox, and deploying off the tunnel so Gary's phone number is always awake.

Because the goal isn't a demo. It's that someone's grandmother can pick up a phone that has been on her kitchen wall for thirty years, and have the modern world actually answer.