Saathi

Inspiration Phones made the world accessible to everyone except the people who can't see them. Blind and low-vision users in India mostly rely on screen readers, which read the screen but don't do anything. Ordering milk on Blinkit, replying to Mummy on WhatsApp or finding a song on YouTube becomes dozens of swipes and double-taps through labels that are often missing.

We wanted something closer to a friend sitting next to you: you say what you want in your own words, in Hindi, English or Hinglish, and it does it with you, step by step. Outside the phone, the same friend should warn you about the chair in your way. That became Saathi (साथी, "companion").

What it does A floating bubble over every app. Tap it or hold Volume-down, then just talk: "is app mein meri help karo", "sabse sasta kaun sa hai", "YouTube pe lofi chalao". Do it together, or do it for me. Saathi reads the screen aloud, then taps, types and scrolls for you, saying what changed after every step. Walk mode. The back camera spots people, vehicles, chairs and doors and warns you by voice and vibration, running offline on the phone. "What's in front of me?" It describes a scene or a photo in plain words. Safety first. Saathi never pays, orders or sends without a read-back and a clear "yes". PIN and password screens are always handed back to the user, and private apps (WhatsApp, payments, SMS) are never sent to any AI. How we built it We started with a Python prototype on a laptop that drove a real Android phone over wireless ADB. It proved the idea, but a laptop is no companion, so we rebuilt Saathi as a native Android app (Kotlin, Jetpack Compose):

Layer What we used Seeing and acting on apps An Android AccessibilityService with one choke point: every tap goes through . a deterministic safety gate Safety gate Rules R1–R11, ported 1:1 from the Python prototype and covered by unit tests Voice On-device speech recognition and offline text-to-speech Walk mode CameraX + ONNX Runtime running YOLOv8n models on-device, about 5–6 . . checks per second Understanding Fixed phrases first, then an AI model for free speech (see below) Floating bubble Overlay windows drawn by the accessibility service, so no "draw over apps" permission

Walk mode estimates distance with the pinhole-camera relation: an object of known real height $H$ that appears $h$ pixels tall, through a lens with focal length $f$ (in pixels), is roughly

$$ d \approx \frac{f \cdot H}{h} $$

away. We combine that with tracking across frames, so only objects that are close and getting closer trigger a warning.

For "do it for me", an AI model plans one step at a time: it looks at the screen's element list and answers with a single JSON action (tap, type, scroll, done, ask). Saathi validates every answer, retries once if it's invalid and then stops, and runs the step through the same safety gate as everything else.

Challenges we ran into

Saathi heard itself. In early tests, Saathi's own voice saying "send" was picked up by its microphone and treated as the user's "yes". A test message really went out. We rebuilt listening around this: the mic never opens while Saathi speaks, anything that echoes Saathi's own words is rejected, and "yes" must be clear and strict. Sends to anyone but the owner's own chat were banned in testing from then on. On-device AI was too slow. We tried Gemma 3 1B on our 2020 tablet. One answer took $\approx 22,\text{s}$ on the GPU and $\approx 58,\text{s}$ on the CPU, against a $6,\text{s}$ budget. Most of the time went to reading the long prompt: cutting it from ~550 to ~70 tokens brought it to $4.5,\text{s}$, but accuracy fell to 3 of 6. For the demo we moved understanding to a small bridge on the laptop, which keeps the API key off the phone and answers in about $1,\text{s}$. On-device AI remains the goal for newer phones. A crash deep in the AI engine. We closed an AI session while the engine was still calling back into it, and the whole app died with a native SIGABRT. The fix: never close a session until the engine has truly finished. The world is noisy. A YouTube Short playing in the background was "heard" as a command. Hands-free listening is now an opt-in setting, and the bubble only acts on the user's own press. Blind-first design is hard. We threw away two visual designs (soothing and glassmorphism) for a minimal, Apple-style "Clarity" design where every screen works by voice and with TalkBack.

What we learned Safety has to be deterministic. The AI may suggest; only the rule-based gate decides. Every rule change came with a test. Latency is a feature. A blind user can't glance at a spinner, so a 2-second answer feels like a conversation and a 20-second one feels broken. Test on the real device, early. Most of our hardest bugs (two-pane WhatsApp on a tablet, rotation, phantom touches, a mic that hears its own speaker) only appeared on hardware. What's next Fully on-device understanding on newer phones, so no screen text ever leaves the device. Sharing walk-mode warnings with step-by-step navigation. Testing with blind users, the people Saathi is for.