Inspiration

A white cane finds the floor. It misses anything at head height, and it can't tell you what is in front of you. We wanted something a blind person could wear that covers both and leaves their hands free.

What it does

Sonar is a baseball cap with an iPhone on the brim.

  • It measures what is ahead with the phone's LiDAR and says "Stop" when something is under a metre away. If you keep walking, two servos at your temples tap hard.
  • It tells you which way is open: "Veer left", "Stop. More room on your right."
  • It names what matters as you get close: a person, a chair, a door.
  • Say "Hey Sonar" or press the button on the cap and ask anything. "What is this man holding?" got back "The man directly ahead is holding a smartphone in his right hand."
  • It speaks up when someone gestures at you, like offering a handshake.

How we built it

  • Phone app (Swift): LiDAR depth split into left, centre and right. YOLOv8n (Open Images) runs on the phone through Core ML at about 7 frames a second, and each box gets a real distance from the LiDAR.
  • Voice: ElevenLabs speaks everything. Warnings are saved on the phone so they play instantly. ElevenLabs Scribe turns the spoken question into text.
  • Questions: two requests go out together. A words-only model answers right away if it doesn't need the camera ("five plus five") or says a short "let me look around". The second one gets the camera picture and the sensor readings, and its answer follows.
  • Cap: an ESP32 with two servos, a button and a floor laser, soldered onto a perfboard and powered by the phone over USB-C. It talks to the app over Bluetooth.

Challenges we ran into

  • The ultrasonic sensor never returned a reading, so we dropped it and used the phone's LiDAR.
  • The breadboard was too big for a hat, so we soldered everything onto a perfboard for the first time.
  • On our first walk it said "Stop" 22 times and named 42 objects in under eight minutes. Most of the afternoon went into deciding what it should not say.
  • The floor laser fired on almost every step when mounted on a moving head. It is on the cap but switched off for the demo.
  • "Hey Sonar" in a room full of people saying "Sonar".

What we learned

A guide that talks too much is worse than no guide. The hard part was not detecting things. It was choosing the one thing worth saying and saying it once.

What's next

Make the floor laser reliable for steps and kerbs, and finish guiding a hand to a door handle, which is built but not yet tested on a real door.

Built With

Share this project:

Updates

Submission history