SightSense

Inspiration

A cane only allows you to understand whether something is there once you touch. It cannot determine what the thing is, how to go around it, or where the door is. We wanted to implement a cane that uses intelligence to decipher objects and their danger in the surroundings of the person, and the ASUS Ascent GX10 made that possible.

What it does

SightSense is an experimental smart guide cane. We use an iPhone on the handle's camera and LiDAR depth to an ASUS GX10, which steers a motorized wheel at the cane's tip through a Raspberry Pi. It finds the clearest direction from depth alone, and it explains actions to do when stopped. to operate the vision language model for assessment of surroundings, you can press the handle's button to enable the assistant and release when finished, in which it answers using the local AI model from the ASUS in under a second.

How we built it

A Swift ARKit app streams frames over WebSockets to the GX10. There, a Python pipeline runs a LiDAR obstacle check, YOLO image classification, and YOLOE object and hazard detection on the GPU. The local agent is a Qwen3-VL 30B used through Ollama for describing, reading, and answering questions. A speech planner decides what is said and in what order. The Raspberry Pi 5 drives the wheel through a motor driver that uses a 25% power cap and a safety cutoff. Most of the parts were 3D printed, and Tailscale is able to tie the devices together.

Challenges we ran into

When we first tested it, the model's first answers received were odd and did not fit the prompt given by the user. Mock tests on real recordings were able to get its sentence understanding to become more intelligent, and a speech planner cut overlapping speech for simplicity. The hardest bug was left and right near obstacles, where the sensing was correct, but anything close was blocked in every direction equally, so the turn was randomized. Turning away from the nearby obstacle raised the correct choice to extremely high accuracies of around 97%.

Accomplishments that we're proud of

All AI and every guidance decision runs through the ASUS Ascent GX10, using no cloud AI, and safety decisions come only from depth. We tested against recordings and the real camera instead of simplyl guessing, which turned issues of the directions being switched into fixes we could prove. We were able to connect an iPhone, a GX10, a Raspberry Pi, a motor, and print the parts into one working loop within a single hackathon.

What we learned

The hard part of assistive technology was not only detecting objects but also deciding what response should be given. We also learned to test against ground truth, as our synthetic tests passed while the real cane gave the wrong direction. Running large models locally is practical and useful on hardware like the GX10.

What's next for SightSense

The future implementation we can do is verify the wheel's direction and speed on the real cane and run supervised walking tests with blind and low-vision users. SightSense is a prototype meant to be used alongside a white cane: it isn't safe for independent travel and doesn't yet handle stairs, kerbs or street crossings, which are the next problems we want to solve.