PathSense

Person holding PathSense smart cane
Assembling components
Electronics inside of handle
ARKit LiDAR Depth Sensor
PathSense smart cane
Electronics inside of handle (2)
Circuit Diagram
Person holding PathSense smart cane (2)
Breadboard connection for Haptic Motors
Main Screen while navigating
Maps and known locations
LiDar

About the project

PathSense started from a simple question: what if a white cane could actively guide, not just detect contact? We wanted to build something that helps blind and low-vision users move more confidently in real environments without forcing them to rely on a screen or constant audio prompts.

Our goal became a smart cane mobility system: an iPhone handles sensing and decision-making, while a smart cane handle provides intuitive left/right haptic guidance. The result is PathSense—an iOS app + ESP32 cane integration that turns spatial awareness into physical steering feedback.

What inspired us

We were inspired by the gap between modern perception technology (LiDAR, on-device ML, realtime navigation APIs) and everyday assistive tools that are still largely passive.

Most other mobility tools are extremely expensive such as:

Service animals
Pre-exisiting smart canes

How we built it

We built PathSense as a modular iOS system in Swift/SwiftUI with dedicated components for sensing, navigation, decision logic, and hardware communication.

Perception: ARKit LiDAR depth frames are split into left/center/right zones to estimate obstacle proximity.
Steering logic: A steering engine blends obstacle-avoidance direction with turn-by-turn route bias.
Terrain safety: A CoreML DeepLab model classifies surfaces and can treat rough/unsafe regions as soft obstacles.
Hardware output: The final steering signal is streamed via BLE to an ESP32 in the cane handle, which drives haptic motors.
Interaction layer: Voice support and a controller-testing mode helped us iterate quickly without always needing full walking trials.

Challenges we faced

Latency and stability: Real-time guidance only works if updates are fast and smooth. Raw sensor data was noisy, so we needed temporal smoothing without making steering feel delayed. Signal fusion tradeoffs: Obstacle avoidance and route-following can conflict (for example, route says right while obstacle says left). Tuning weights and clamping behavior was critical. Hardware reliability: BLE dropouts and packet timing had to be handled gracefully so the cane behavior stayed predictable. Human factors: Haptics must be clear but not overwhelming. Small changes in intensity, cadence, and direction mapping had a big effect on usability. Hackathon constraints: We had limited time to integrate mobile sensing, ML inference, mapping, and embedded firmware into one coherent experience.

What we learned

This project taught us that accessibility systems are as much about interaction design as they are about algorithms.

We learned: to design around user cognitive load, not just model accuracy, to build resilient real-time pipelines across phone + embedded hardware, to iterate quickly by decoupling subsystems and testing each one independently, and to treat safety-oriented edge cases as first-class product requirements. Most importantly, we learned that meaningful assistive tech can be built from consumer hardware when software is thoughtfully integrated.

Where this can go next

PathSense is a strong prototype, and we see clear next steps: user studies with blind/low-vision participants, adaptive personalization of steering sensitivity, richer obstacle semantics (moving vs static), offline-first routing and fail-safe modes, and miniaturized production hardware for daily use.