Pawthfinder

Inspiration

About a million Americans are blind, and millions more use wheelchairs, canes or walkers. For them a city is an obstacle course: curbs without ramps, crosswalks without audio signals, scooters on the sidewalk and, in Miami, streets that flood. Cities fix this one corner at a time. A talking crosswalk adds about $3,600 per intersection and one curb ramp costs about $2,500. Most places aren't there yet. A self-driving car can take someone across the city, but the trip isn't finished until they're safely in the seat, and for many riders the hardest part is the walk from the door to the curb. So we asked: what if the safety check traveled with the person instead of waiting for the city?

What it does

The whole trip, door to seat. Pawthfinder plans the ride and the walk as one trip. SafeRoute uses public data (Google Routes and Elevation, FDOT crash records, Miami-Dade 311 flood reports, NOAA tides and NWS alerts) to choose the safest way through the city and the safest curb for a Waymo pickup, and Waymo's open-source Waymax simulator checks the car can pull in there. Our robot guide dog covers the last stretch on foot, up to the passenger door, where our vision model finds the handle.

A robot guide dog, real and simulated. The rider never types or opens a chat window: they stand at the curb, and the robot tells them when it's safe and walks them across. On a real MechDog, Google's Gemini reads the pedestrian signal through a camera and tells a solid red hand from a flashing one. On WALK the dog crosses on its own, stops by sonar at the far side, and talks the rider through each step in an ElevenLabs voice. It works with the signal already at the crosswalk, so nothing has to be installed. In simulation, the same dog leads a blind rider on real Miami streets, planning from its own camera. It never leads the rider off a curb.

Obstacle course: 112 m down Biscayne Blvd around real bus stops and FDOT work zones to a waiting Waymo.
Flooded sidewalk: its route runs through a real Miami-Dade 311 flood report. It sees the water, replans around the block, and gets the rider there dry.
Busy crossing: at a real five-lane crossing, our YOLO model reads the walk signal from the dog's camera. The dog only starts when there's enough time to finish at a cane user's pace.

SafeRoute, a safety-first map. Map apps pick the fastest route. SafeRoute scores every option against tens of thousands of real hazards (crashes, flood reports, construction, live police crashes, NOAA tide and rain, NWS alerts, hurricane surge zones) and picks the safest. Time and battery only break ties, and Gemini explains the pick in plain English.

How we built it

Real robot: a MechDog (ESP32) running our own MicroPython bridge over USB, which stops the dog if commands stop for 1.5 seconds. A laptop agent sends Gemini Flash-Lite the full camera view plus three zoomed frames 0.4 s apart, about every two seconds, and gets back JSON: walk, flashing hand, solid hand or none, with any countdown. The dog crosses on go, waits on a solid hand, and ignores answers older than five seconds. Two Flash-Lite models run on a staggered schedule to stay inside rate limits.
Escort agent: Gemini gets the rider's goal and what our YOLO model sees, and returns the dog's next move as JSON (walk or stop, stride, turn, speed, and a line to say). Walk and stop signals are handled by fixed rules first, so a stop never waits on the network.
Voice: ElevenLabs. Every line is tied to what the robot just did, from "I'm watching the crossing signal" to "Walk signal detected! Crossing now, stay with me." Lines are generated ahead of time, so each plays the instant its event fires.
PyBullet worlds built from OpenStreetMap (real lanes, curbs, ramps, buildings) plus Miami hazard data. The dog turns its depth camera into a height map, finds curbs, holes and water, and replans with A* every step.
A custom YOLO26 model with a map-guided crop reads a real-size walk signal from 18 m away. It also knows the Waymo vehicle, its door handle and an open door.
Our own route planner on the OpenStreetMap road graph, with the final pull-in driven in Waymax.
SafeRoute: Node.js on Vercel with the Google Routes, Elevation and Maps APIs, a Pareto ranking with an EV energy model, and a recorded-demo fallback clearly labeled "not live".
Accessible by design: keyboard and screen-reader support, 44 px touch targets, text versions of every chart, and burned-in subtitles in our demo video.

Challenges we ran into

A low camera can't see over a curb, so drop-offs look like missing ground. We treat "ground we should see but can't" as a possible drop.
The vision model couldn't read a real walk signal from across the street until we cropped around its mapped position.
The real robot's own fall and obstacle stops fired on every step's shake and false sonar echoes, cutting the legs mid-stride. We rewrote both in its firmware.
Today's $299 MechDog walks about 0.25 m/s, too slow for that crossing. The simulation uses a cane user's 0.9 m/s and says so.
Messy city data and Google's daily limits pushed us to add caching and recorded demos.

Accomplishments that we're proud of

Every scene uses real streets, real flood reports and real crash history. The dog decides from what it sees, not from a script, and the real robot crosses on a real walk signal. Anything illustrative is labeled on screen.

What we learned

The safest route is often only seconds slower, and nobody shows people that choice. Accessibility isn't one feature: curbs, signals, water and the ride all have to work together.

What's next for Pawthfinder

Rubber feet and a faster gait so the real MechDog can walk outdoors, the simulation's planner on the real dog, haptic guidance to go with the voice, wheelchair routes around missing ramps and steep slopes, and more cities.