Inspiration
43 million people worldwide are fully blind. Few people can help them every day, and a paid guide costs about $15 an hour. Most assistive apps either can't describe the surroundings or send the camera feed to a server. We wanted an assistant a blind person can use alone, anywhere, in their own language.
What it does
EYE (λκΈΈ) is an on-device scene assistant in English and Korean. It works without internet.
- π Look around β turn once and hear what is around you, where it is and what colour it is: "3 blue chairs in front; a black laptop on your right."
- π Find β say "find my bag": beeps speed up as you point at it, and the phone vibrates when it's straight ahead.
- π€ Saved people & objects β save a face with one turn of the head; EYE names them during scans.
- π Reader β reads QR codes, barcodes and printed text aloud.
- πΆ Walk mode β ARCore depth warns about walls, steps, stairs, drops, people and cars.
- π Places β "save this place as home", "take me home"; without internet it still points the way.
- π Voice control β wake word "Eye" / "λκΈΈμ"; the app goes silent the moment you speak.
How we built it
- Kotlin + CameraX Android app
- MediaPipe EfficientDet-Lite2 for object detection
- Our own sensor fusion: compass heading + camera field of view give every object a direction in a 360Β° scan
- ML Kit + FaceNet-512 (TensorFlow Lite) for faces, MobileNetV3 embeddings for saved objects
- ARCore depth for walk mode
- Offline speech recognition and text-to-speech with an English/Korean command parser
Challenges we ran into
- Counting: in a 360Β° turn the same chair appears in dozens of frames, so we use the highest count seen in any one frame, never the total.
- Hearing itself: the microphone picked up the app's own voice, so we added a wake word and silence every sound when the user speaks.
- ARCore crashes: it crashed when restarting in the dark or while the camera was hidden. We changed when it restarts, and EYE now says when it's too dark.
- False alarms: walls were mistaken for stairs, and elevators triggered floor-change alerts, so we rewrote the depth rules.
Accomplishments that we're proud of
- On-device AI β camera images never leave the phone
- Fully bilingual voice control and speech
- 360Β° scan with direction β we found no open-source project that combines detection with compass heading
- Tested on a real street, with real blind people
What we learned
- For a blind user, every extra word costs attention. Saying less, at the right moment, beats detecting more.
- A real street finds bugs a desk never shows.
- Running ML on a phone is mostly tuning: thresholds, model size, and falling back from GPU to CPU.
What's next for EYE
- π°π· Korean OCR for signs and menus
- π¦― Testing with blind users and improving EYE from their feedback
- π¦ Crosswalk and bus-number detection
- π’ Indoor navigation where GPS fails
- π§ Earbuds and smart glasses support, then a Google Play release
Log in or sign up for Devpost to join the conversation.