breadcrumb

Inspiration

A map can get you to a building. Finding the right room is another problem.

“Turn left” is not enough when there are three hallways, two staircases, and a door with no room number. Someone who knows the place would say, “Go through the door under the red sign, then take the elevator to the third floor.”

We built Breadcrumb to turn that local knowledge into routes other people can follow.

What it does

Breadcrumb is a mobile-first web app for teaching and following indoor routes.

A creator records a walkthrough and uploads it. Gemini turns the video into a draft with landmarks and directions. The creator checks the steps, makes corrections, and approves the route.

Visitors can ask for a destination through iMessage, get a route link, and open the guide on their phone. They can follow checkpoint instructions or use camera-based guidance with spoken directions and captions.

If a route is missing, visitors can post a bounty. Someone who knows the way can record it and submit it for a reward using sandbox money.

How we built it

We built the app with Next.js, React, and TypeScript, using shadcn/ui and Motion for reusable components and animation.

Each integration serves a specific part of the experience:

Google Gemini: Converts walkthrough videos into editable route drafts. Gemini Live uses camera frames and route context to provide spoken guidance and captions.
xAI Grok: Matches natural-language destination requests to our approved route catalog. We validate the returned route IDs and build the links in code.
Photon Spectrum: Connects the experience to iMessage. Users start by messaging where they want to go and receive route options and guide links.
ElevenLabs: Reads approved checkpoint instructions aloud in English and Spanish.
Capital One Nessie: Supports route bounties through sandbox accounts and payouts. A requester approves the submitted route before releasing the reward.
Cursor: Supported development through code review and checks of the voice integration (156,128 additions, 1,116 deletions).

We also added video compression to reduce upload size. Route data and media are stored on the host computer for this prototype.

Our GoDaddy domain: iamlostwheredoigo.us. Speaks for itself.

Challenges we ran into

Indoor directions need more detail than a sequence of turns. We had to represent doors, signs, floors, stairs, and elevators in a form that people could review and the app could use.

Phone recordings also introduced large files and different video and audio formats. Compression needed to preserve useful details and narration.

Another challenge was keeping generated output grounded. Route drafts require human approval, Grok’s matches must reference existing routes, and live guidance remains advisory.

With several people working at once, we used shared API contracts, separate areas of ownership, and pull request reviews to keep the parts compatible.

Accomplishments that we’re proud of

We connected route creation, destination discovery, and guidance in one web app.

We tested video extraction and live guidance with real Gemini API calls. We also built the iMessage entry point, approved-text speech, and a route bounty flow.

The bounty board gives the project a way to grow: people can request the routes they need, and people who know the place can contribute them.

What we learned

Good directions depend on recognizable details. “Use the door under the red sign” can be more useful than a distance or a turn.

We also learned that generating instructions is only part of the work. People need to review them, visitors need a clear way to find them, and the app needs to handle uncertainty.

Our tests separate simulated responses from real API calls and physical route trials. Gemini extraction and Live passed real API tests. Photon, Grok, ElevenLabs, and Nessie are implemented and tested with simulated responses; live verification is still pending.

What’s next for Breadcrumb

Our next step is to test complete routes with people who have never visited the location. We want to measure whether they reach the destination, where they hesitate, and which instructions need work.

We also plan to verify the remaining integrations live, improve mobile media support, and move beyond storage on one computer. After the phone experience is reliable, we want to explore Meta Ray-Ban glasses for hands-free guidance.