Scout

Logo Banner
Title Slide
Intro
Methods
Methods
Results
Results
References
Obstacle Identification + Detection System
LiDar Depth for Obstacle Detection
Detection + Direction Logistics Log
Audio Cue Customization Setting

Inspiration

Our team built Shepherd, a motorized LiDAR cane, before this event. It worked, but it was one more expensive device to buy, charge, and carry. Commercial smart canes cost hundreds of dollars-- the common WeWalk cane costs ~1250$-- and they lock users into a single tool. Most people with low vision already own a phone with a depth sensor and already trust their cane, walker, or guide. We wanted to make what they already carry smarter, not sell them something new.

What it does

Scout turns a LiDAR iPhone into a navigation layer for any mobility tool. Today it clips to the chest; the same software is built to move to a cane, a walker, or glasses.

Detects obstacles, drop-offs, curbs, and stairs ahead with LiDAR, and finds the clearest way through.
Speaks in short, kind sentences through the phone speaker: "There's a door just ahead, keep a little to your left." Safety cues stay instant: "Stop." always plays first, and urgency is felt as vibration.
Names what is in the way (people, doors, desks, chairs, poles, vehicles) on the phone, with no network, and points out open doorways. Gemini names anything the phone cannot.
Guides to a destination with walking directions, or runs in passive mode with no destination at all.
Works offline for everything safety related, with three voice levels (Minimal, Normal, Chatty) and a live camera view showing exactly what it detects.

How we built it

We built Scout in gated phases. Each phase had a pass condition before the next one started.

Skeleton. A SwiftUI app that captures ARKit depth on a background queue at 30 fps, with every threshold in one tuning file.
Record and replay. Walks are recorded and replayed through the same pipeline, with a hash that proves two replays give identical output. This let us debug a walk without walking it again.
Point cloud and floor. Each depth pixel is projected into 3D. Scout learns the floor height, then sorts points into floor, obstacle, drop-off, and overhead, drawn live over the camera image.
Depth grid and planner. The points are flattened into a top-down grid around the walker. The planner marches walking-width corridors through the grid and picks the clearest heading, with smoothing so guidance does not flip left and right.
Cues. One arbiter decides what is said and when, using bundled ElevenLabs clips for instant, offline speech and haptics for urgency. Stop interrupts everything; nothing repeats every frame.
Passive mode, Siri, and verbosity, tested toward a walk with our blind tester.
Guided mode. Apple MapKit walking routes plus a north-aligned heading.
Identification. Apple Vision finds people and dogs, a bundled YOLOv8 model (Open Images, 600 classes) finds doors and furniture, and the depth grid finds poles and doorways. A name is only given to something LiDAR already sees, so a wrong or missing name can never hide an obstacle. Gemini names leftovers while they are still more than 2.5 m away, so the answer arrives in time.

Challenges we ran into

The depth grid was the hardest part. A chest camera never sees the floor right in front of the walker, depth is noisy at object edges, and a few stray pixels can become a phantom wall. We had to learn the floor height before trusting anything, require evidence across several cells before calling something an obstacle, and remember the map as the user turns so obstacles do not vanish the moment they leave the frame.

Identification was the second. Our first naming build detected obstacles but spoke no names, and we could not tell which stage was failing, so we added an on-screen status line that reports each step from detector to spoken cue. Gemini brought its own limits: one to two seconds per photo is far too slow for a "Stop," and the app had to hold a key without it ever touching our code. We asked Gemini early, only about distant obstacles the phone could not name, and kept the key on the device.

Accomplishments that we're proud of

Depth-only obstacle, drop-off, and stair detection that runs offline at walking speed.
Identification that is honest by design: the camera names what LiDAR sees and never decides what is safe.
Doorway detection from depth alone, which no off-the-shelf detector gave us.
Speech that sounds like a person helping rather than a machine reading coordinates.

What we learned

We learned how LiDAR depth becomes geometry: projecting each pixel into 3D, finding the floor plane, and reasoning in the walker's frame rather than the camera's. We learned that an occupancy grid searched with walking-width corridors is far more stable than reacting to raw depth. Most of all, we learned that for a blind user timing matters more than wording. Every extra word is walking distance, so the most important cues have to be the shortest.

What's next for Scout

Next is making Scout truly universal: 3D-printed mounts for canes, walkers, and glasses, each with stabilization and its own calibration for camera height and swing. We will use walks recorded with blind testers to tune our thresholds and fine-tune the detector on obstacles seen from a walker's height. Then we will test with more people in more places, and measure missed obstacles and false stops instead of relying on impressions.