AcousticMaps

Joy Wang · Sophia Lu

Inspiration

Sophia is a poor navigator, and looking at a map while walking doesn't fix that. For blind and low-vision walkers a map on a screen isn't an option at all.

We wanted walking directions that arrive by voice and touch: the app says when to turn, notices when you drift off the route, and answers questions, with the phone in a pocket.

What it does

Voice input. Tap anywhere and talk. "Take me to Malott" and "somewhere I can get coffee" both work.
Spoken directions. Short instructions using left, right, ahead and behind.
Haptics. Two pulses for a left turn, three for a right, one long buzz when you're off route.
Corrections. If you walk the wrong way, it says which way the path is and how far.
Questions. "What's around me?" names nearby buildings and where they are.
Blob. Our mascot sits in a compass and shows the trip's state on screen.

Blob's six states: listening, thinking, turn right, turn left, off route, arrived

How we built it

The iPhone app sends position and audio to a FastAPI server, which uses Grok, Gemini and OpenStreetMap and sends back what to say, the haptic and the route

iPhone app

Swift and SwiftUI, deliberately thin. It sends position once a second, streams microphone audio, speaks replies, and plays haptics. It keeps working with the phone locked in a pocket.

Server

Python and FastAPI, where all the logic lives. Every second it runs an ordered list of checks (weak GPS, arrived, off route, turn now, wrong way, turn ahead) and the first one that fires is what you hear.

Grok speech-to-text

Audio streams from the phone to our server and on to Grok, which transcribes live and decides when you've stopped talking, so there is no stop button.

Gemini agent

Handles anything open-ended. It runs function calling in a loop of up to 4 rounds within 8 seconds, over six tools backed by real data:

search_places: Cafés, libraries and so on near you, from OpenStreetMap
plan_route: Distance, turns and minutes to a place, without starting
start_navigation: Starts the trip and sends the route to the phone
cancel_navigation: Ends the trip
trip_status: Distance left, the next instruction, and which way back to the route if you're off it
describe_surroundings: Named buildings within 60 m, with left, right or behind directions

Gemini also rewords our instructions so they sound natural.

OpenStreetMap

Walking paths, places and landmarks, cached so a trip doesn't wait on a slow API.

Testing

A fake phone walks any route with GPS noise, detours and reversals, and we replay recordings of our real walks through the server.

Challenges we ran into

Navigation

Curved paths. Straight-line distance is wrong on a bend. We measure distance along the route instead. On a quarter-circle test route of 20 segments, that gives 78.5 m where the straight line gives 70.7 m.
Knowing a turn was passed. Our first rule was "the distance to the turn stopped shrinking." It broke when someone cut a corner, when a route passed the same spot twice, and when GPS jumped. We now track progress as distance along the route, which only moves forward.
Repeated GPS readings. The phone sends an update every second, but GPS doesn't always have a new reading, so the phone re-sends the last one. The server compares each reading's timestamp and skips repeats, so one bad fix can't count several times.
GPS that is plainly wrong. Readings with poor accuracy are ignored for tracking, and a correction needs three bad updates in a row before the app speaks.
Junctions with similar paths. Where two paths leave a junction at close angles, "turn left" doesn't say which one. The instruction names a landmark and uses words like "slight left," and the same wording guides someone back onto the right path if they take the wrong one.

iPhone

Connecting a physical iPhone to Xcode.
Getting app signing, developer trust and permissions working.
Reading live location, heading and walking speed.

Voice

Streaming audio from the microphone to our server and on to Grok.

Accomplishments that we're proud of

The full trip works: speak a place, pocket the phone, and walk there.
Gemini handles open requests and fixed rules handle guidance, so directions never wait on a model.
We collected our own data. The server logs every update from the phone, so each real walk is saved as a recording.
A simulator that replays those recordings, so we can test a walk without leaving the building.
We corrected the guidance from real walks, not guesses: when a recording showed a wrong instruction, we replayed it until the fix held.
Blob, drawn by hand.

What we learned

Real GPS behaves differently from a simulator, and we only found the problems by walking the routes.
Which parts suit a language model and which are better as a few if statements.
Writing for someone who can't see the screen: no visual references in speech, and every state needs a sound or a haptic.