CaneSight

Inspiration

We wanted to explore how technology could add useful information to a familiar mobility tool. CaneSight combines a white cane with depth sensing, a camera and accessible feedback to help people with vision loss understand their surroundings. Our goal is to supplement their awareness while preserving their independence and established mobility techniques.

What it does

CaneSight is a smart-cane research prototype built around an iPhone with LiDAR and a Raspberry Pi camera.

Our iPhone app displays depth information across three image regions, with large controls, VoiceOver labels, spoken status and manual audio cancellation. Missing or outdated readings become Unknown.

The Pi supports camera uploads and a physical-button spoken menu. We also built software for bounded vibration cues and an isolated AI scene-description service. These components are progressing toward a complete connected system; the live phone-to-motor flow still needs validation.

Pedestrian-signal and countdown recognition remain replay experiments. CaneSight does not determine when it is safe to cross a street.

How we built it

We built the iPhone app with Swift, SwiftUI, and ARKit, using confidence and coverage filtering to process LiDAR depth measurements.

Our Raspberry Pi Zero 2 W provides camera and button components. Python edge software handles input, cancellation and a two-channel haptic scheduler.

The backend uses Python, FastAPI, HTTPS and authenticated WebSockets for paired communication and bounded processing. We added a pinned YOLOv8n baseline for generic object detection, separate signal replay tools and a restricted Gemini description service.

We also implemented consent-controlled event logging with Tiger Data/PostgreSQL and verified synthetic storage and deletion behavior against a real database. GitHub reviews, automated tests and handoffs helped us develop the components in parallel.

Challenges we ran into

One of our biggest setbacks was breaking the camera’s copper ribbon cable. With limited replacement hardware available, we sourced a camera from old hardware and adapted our setup to keep building.

Limited access to the Mac, LiDAR phone and Pi also shaped our workflow. We used portable tests and clearly labeled replay fixtures to continue software development while recording device results separately.

Another challenge was handling uncertainty. A connection can stay alive while its sensor readings become stale, and stopping a speech process does not necessarily stop audio already buffered by a Bluetooth speaker.

We also learned that generic traffic-light detection is very different from recognizing pedestrian symbols, reading countdowns or establishing flashing over time.

Accomplishments that we're proud of

Running our native app on a physical LiDAR iPhone with live depth readings, accessible controls and Stop-to-Unknown behavior.
Building a Pi button-and-spoken-menu workflow, with operator-reported clear speech and prompt cancellation during controlled tests.
Creating authenticated backend components that preserve observation identity and original timestamps.
Developing bounded haptic-control software and regression tests for disconnects, expired data, replay and cancellation.
Recovering from broken hardware by repurposing an older camera.
Keeping unsupported observations explicitly unknown instead of overstating the prototype’s capabilities.

What we learned

We learned that assistive technology requires more than a model prediction. Timing, accessible controls, clear feedback and predictable failure behavior all matter.

We also learned to distinguish software correctness from physical performance. Replay tests helped us verify logic, while device testing exposed separate issues involving app lifecycle, audio routing, sensor behavior and cancellation.

Working across hardware, mobile development and backend systems taught us to define interfaces early and record exactly what each test demonstrated.

What's next for CaneSight

Our next milestone is a complete, measured phone-to-backend-to-Pi demonstration with verified capture timing, interruption recovery and bounded vibration output. We also need to finish electrical, motor-mapping and physical audio-buffering checks.

For vision, we plan to collect and license more real footage, review annotations and separate training and evaluation data by location and recording session. That will support custom pedestrian-signal models and further evaluation of countdowns, beacons and vehicle-motion observations.

We also want feedback from people with vision loss and orientation-and-mobility specialists to improve cue clarity, controls and usefulness. CaneSight remains a research prototype, with future demonstrations limited to controlled settings rather than road crossings.