BreadCrumb

## Inspiration

Our friend is visually impaired. Crowded buildings, big parking lots and unfamiliar trails are hard for him, because most of the clues that tell you where you are come through your eyes: signs, doorways, the shape of a path. Map apps help outside on a street, but they fall apart inside a building or off the road, and they expect you to look at a screen.

We wanted to build something he could use without looking. You tell it where you want to go, it looks around for you, and it talks you there one step at a time.

## What it does

Breadcrumb is a helper in your phone for when you are lost somewhere without a map.

1. **You say what you are looking for.** "The exit." "A road." "The car park."
2. **You turn slowly in a circle.** The phone takes 8 photos as you turn. Each time you have turned a bit more, it buzzes and takes the next one, and it notes which way it was facing, like a compass.
3. **Gemini looks at the photos and thinks like a person would.** "There is a doorway with an exit sign. There is a wall over here. That hallway leads toward the busy part of the building." Then it picks the most promising way.
4. **It tells you out loud where to go.** "You're at a fork. Paths go left and right. Take the left path. Turn left, then go straight." As you turn, it keeps talking: "slightly left… straight ahead, go now," and buzzes when you are lined up.
5. **You walk, then do it again.** Every place you stop becomes a dot on a map. Places where the path splits are marked as forks, and every choice is drawn: the way you went, the ways you haven't tried, and the ways that failed.
6. **It remembers.** It keeps a few small photos of every place you stop. If you walk in a circle and end up somewhere you have already been, it notices: "You've been here before. The left path was a dead end." It will not send you down the same wrong path twice.
7. **You stay in charge.** If you know a path won't work, one tap on "Not that way" crosses it off, and it picks again.
8. **It knows when you have made it.** When the thing you asked for shows up in the photos, it says "You have reached the exit."

## How it works (the simple version)

Think of it like leaving breadcrumbs in a forest, except the phone remembers them for you.

**Pointing the right way.** When the phone takes a photo, it knows which way the camera faced. But an exit might be on the left side of that photo, not the middle. So Gemini also tells us *where across the photo* the exit is, from 0 (far left) to 1 (far right). We turn that into a direction:

$$
\text{direction} = \text{photo heading} + \arctan\big((2x - 1)\cdot\tan(\tfrac{\text{FOV}}{2})\big)
$$

where $x$ is the exit's position in the photo and FOV is how wide the camera can see. For a 60° camera, an exit at the very right edge ($x = 1$) is $30°$ to the right of where the photo was aimed.

**Remembering the way back.** When you walk into a new place, the path you just came down is behind you. We label it "the way back" so the app never mistakes it for somewhere new.

**Deciding what to do.** Gemini judges which paths look promising, and the phone checks its choice against the map. It will never pick the way back, never pick a path that already failed, and if every path here has been tried, it works out the shortest walk back to the nearest fork that still has an untried path.

## How we built it

- **Gemini** reads each set of 8 photos along with the map so far, and answers with structured JSON: what this place looks like, every way out it can see, whether this is a place we have been before, and which way it recommends.
- **The phone's compass and motion sensors** measure which way you face and count your steps, so each new place lands on the map roughly where you walked.
- **The browser's speech and vibration** do the talking and buzzing, so nothing requires looking at the screen.
- **A small Express server** checks each request and talks to Gemini. The map, the memory and the decisions all live on the phone.
- **A map drawn on a canvas** shows forks as numbered diamonds, with each path marked as walked, untried, failed or chosen.

## Challenges we ran into

- **The arrow was always a little off.** At first we pointed at the middle of whichever photo showed the exit, which could be 30° wrong. We also found the compass reading was only correct when the phone was held perfectly upright. Fixing both meant doing real geometry on the phone's tilt sensors.
- **Turning slowly broke it.** Our first version took photos on a timer, so if you turned carefully, which is exactly what you should do, it ran out of photos before you finished the circle. Now it waits for you to turn instead of waiting for the clock.
- **Bad memory.** Every stop saw "the way I came" and thought it was a brand new path, so dead ends were never detected. Recognising a place from a different angle was hard too, so each place now keeps photos facing several directions.
- **Trusting the AI too much.** Gemini sometimes picked a path it had rated lower than another, or suggested going back the way you came. We learned to let Gemini do what only it can do, which is read the photos, and to let plain rules handle the rest.

## What we learned

- The hard part of this problem is judgement about pictures: is that gap in the trees a path? Is this the same hallway as before, seen from the other side? That is exactly where Gemini shines, and a normal program cannot do it.
- Everything else works better as simple, testable rules. We wrote tests for the map, the memory and the decisions, and they caught bugs we would never have found by walking around.
- Designing for someone who cannot see the screen changed everything. Directions have to be about *your* body ("turn left"), not about the picture ("the path on the right of the photo"), and every important moment needs a sound or a buzz, not just something on screen.