Kenshi

Inspiration

At a teammate's high school, blind and low-vision students practiced getting around by walking with someone who told them where to go. They were that person, helping visually impaired people back to their seat or out of a room. It showed them that they couldn’t do it alone, because nothing could tell them where the thing they were looking for was.

Apple’s Find My guides you to a lost item by telling you which way to go and how close you are. We wanted to build something like that for blind and low-vision people finding things like an exit, a door, or a chair, because they have no way of doing it themselves.

What it does

Kenshi helps blind and low-vision users find a specific thing and guides them to it. You say “find the exit” or tap a quick-pick target like Exit, Door, Elevator, Stairs, Restroom sign, Chair, Trash can, or Person.

If the target isn’t in view, Kenshi asks you to turn slowly and takes photos as you turn. Google Gemini looks for the target in those photos, and Kenshi remembers which direction it was in. Then it guides you with:

Spoken directions like “Turn right” and “Exit, about 3 meters.”
Audio pulses in your AirPods that speed up as you get closer and shift toward the side the target is on.
A screen with a large arrow and distance that turns green when you’re facing the target.
A line on screen showing a path around obstacles.

Kenshi is meant to be used alongside a cane, not instead of one. Obstacle alerts keep running during Find, urgent ones interrupt it, and it never says the way is clear.

How we built it

Web app on the phone: a YOLO object detector runs in the browser, along with tracking, distance estimation, and the audio engine. Obstacle alerts work without internet.
Gemini for finding targets: a Vercel function sends up to 8 camera frames to Gemini in one request. It returns which frame shows the target, where it is, a distance estimate, and any sign text. If it’s unsure, it reports “not found.”
Heading memory: the phone’s gyro tracks which way the camera is facing, so Kenshi can keep guiding after the target leaves the frame.
A state machine coordinates voice commands, Gemini, heading, and YOLO. For objects YOLO already recognizes, like chairs, it skips Gemini.
Path line: a path estimator ported from a teammate’s OpenCV code.
Home-screen install: it launches full screen from an icon, with no App Store needed.
We built it with Claude Code, one step at a time, with tests for each module.

Challenges we ran into

No haptics. AirPods have no haptic motor, and iPhone Safari can’t vibrate the phone. We used audio pulses instead, and gave “behind you” its own sound because stereo alone can’t tell front from back.
Gemini rate limits. We send one request per scan and limit re-checks to once every six seconds, using YOLO tracking in between.
Privacy. On Gemini’s free tier, Google may use what’s sent. We tell the user before the first Find and say a real launch would need the paid tier.
Time. We changed our design with about six hours left.

Accomplishments that we're proud of

We replaced Find My’s haptic feedback with an audio system that works through AirPods.
Safety rules are part of the design: it says “not found” when unsure, urgent alerts take priority, and it never says the way is clear.
It runs on a phone people already own and installs from a link.

What we learned

Plan around what a platform can’t do. AirPods have no haptic motor and iPhone Safari can’t vibrate, so we rebuilt Find My’s feedback as sound. Working with that limit shaped the whole design.
For a blind user, audio is the interface. The screen is there so sighted people can follow along, but the spoken directions and pulses are what the user actually relies on.
Be honest about limits. Find needs internet, the free Gemini tier has data terms we had to disclose, and distance to non-COCO targets only updates when Gemini re-checks. We designed around each one and said so in the app.

What's next for Kenshi

Test with blind and low-vision users through a local blindness organization. Move to Gemini’s paid tier for a real launch. Build a native app for haptics and better sensors, then support camera glasses. Improve the path line on patterned floors and in hard shadows.