Cane-Eye

Obstacle avoidance helping navigate out of corners
Ultrasonic distance sensors helping maintain centering in a hallway
Closeup of the device mounted on a cane
CAD screenshot

Inspiration

A team member met a blind person recently and spent a lot of time thinking about blindness, eventually coming across the idea to integrate modern technology into the classic white cane. The idea of integrating obstacle detectors and AI guidance came up, and it was chosen as a valuable candidate for this hackathon because of its applicability to real-world issues.

What it does

The device attaches to a white cane often used by visually impaired individuals. Using an array of ultrasonic sensors and dual haptic motors, the device provides tactile feedback to the user indicating the distance and direction of nearby obstacles. Additionally, the device houses the user's cell phone, and uses its camera in conjunction with any OpenAI interface--compatible vision language model (VLM), such as Claude 4.5, to provide descriptions of surroundings and specific objects at the push of a button, as well as proactive warnings about surroundings when applicable. For example, if a user is approaching a crosswalk, the system will automatically identify it and tell the user for their safety. Feedback is spoken to the user through the phone, allowing easy interfacing with headphones or earbuds.

How we built it

The core of the device is a 3D-printed frame carrying three ultrasonic sensor modules, an ESP32, haptic motors, a keypad, and a phone mount.
The device's circuitry was constructed by soldering together individual custom "modules" on perfboard, including a sonar module, a microcontroller module, and a motor control module.
The ESP32 microcontroller runs C++ software that provides an I/O interface with the device hardware through JSON sent over WebSockets.
Asynchronous Rust code handles interfacing with the JSON API, the logic of the device operation, and querying vision APIs.

Challenges we ran into

An ultrasonic distance sensor was defective, requiring reassembly of the entire sonar module.
VLM APIs (Claude, etc.) seemed to provide very different responses to different devices.
Integration of software with the target phone platform presented difficulties not present in laptop testing, including network certificate issues and linking problems.
A hygrometer was considered for puddle/wet floor detection, but it was unable to work with the long wire length required to place it at the end of the cane.

Accomplishments that we're proud of

The project makes use of custom-made haptic motors for providing the user with tactile feedback.
The CAD model was quite good. Its modular design and quick turnaround time meant that we had a lot of flexibility in later design.

What we learned

Make sure to test individual hardware components, especially verifying validity of circuit connections, before committing to assembly.
We learned more about how the paradigm of asynchronous code operates.

What's next for Cane-Eye

The device may be adapted to work via Bluetooth instead of WebSockets to make it more simple for a consumer to use.
Additional sensors may be integrated, such as the aforementioned hygrometer.
The warning system could be made more robust. For instance, it could use accelerometer information from the phone to decide when to search for dangers, instead of being restricted to a fixed time interval.