Inside

Inspiration

Maps get us to a building, but finding our way around inside is still surprisingly awkward. Where is the café? What can I do in the games area? How far apart are they?

I wanted to make exploring a building feel like talking to someone who knows it. Ask a question, see the place, and follow an arrow through your phone’s camera.

What it does

Inside turns a phone recording of a building into an interactive voice guide with indoor AR navigation.

On desktop, you can talk naturally with GPT Live 1, browse photographs of different places, and ask questions such as:

• “What’s in this building?”
• “Show me the games area.”
• “What can I do there?”
• “How far is the café from the games area?”

On mobile, the app combines visual localization with WebXR tracking to guide you toward a saved destination. The interface keeps the camera view open: a direction arrow, distance in meters, a light-blue map outline, and microphone and exit controls. The voice assistant runs in the background.

The current prototype reports straight-line distances and directions. It does not yet calculate obstacle-aware walking routes.

How I built it

The project starts with a browser-based scanner. As I walk through the building, it captures images alongside WebXR camera poses and uploads them incrementally to the server. This avoids keeping an entire building recording in the phone’s memory.

The recorded data provides a shared coordinate system for saved places. Reference images support visual localization when returning to the building, while WebXR tracks movement between localization updates.

GPT Live 1 handles the spoken conversation. Building lookups, photo selection, and distance calculations run through deterministic application tools. Distances come from recorded coordinates rather than the model guessing them.

The desktop guide uses JavaScript, HTML, and CSS, with Cloudflare Workers and R2 handling backend requests and stored assets.

Challenges I ran into

Getting a stable coordinate system was only part of the problem. Returning later requires matching the camera view to the recording and aligning the new AR session with the saved map.

Motion blur, viewpoint changes, and temporary tracking loss made this harder. I added on-screen diagnostics and worked on matching and capture behavior instead of simply increasing the recording frame rate.

Another challenge was making voice interactions reliable. Correctly hearing “café” is not enough if the application then opens the wrong photographs. Transcript and tool-call logging helped identify these failures, and deterministic lookups made the behavior easier to inspect and debug.

The interface also needed a rethink. Early versions covered too much of the camera with panels and diagnostics. The final mobile design makes navigation the focus and keeps most of the screen unobstructed.

Accomplishments that I'm proud of

I scanned an entire building with a phone and connected that recording to a working voice guide and AR navigation demo.

The project brings together image capture, spatial tracking, visual localization, stored building knowledge, voice interaction, and a minimal camera interface.

I'm especially proud that the assistant can discuss places and retrieve their photos while spatial calculations remain grounded in actual recorded coordinates.

What I learned

Tracking your movement within an AR session and recognizing your position in a previously scanned building are different problems. Reliable indoor navigation needs both.

I also learned how much observability matters. Live tracking statistics, transcripts, and tool-call logs made it possible to separate camera problems, localization failures, and conversation bugs.

Finally, a more capable AR app does not need more interface. Removing visual clutter made the experience feel much more natural.

What's next for Inside

I want to improve localization across lighting and viewpoint changes, add walkable routes around walls, and make building setup easier for someone using it for the first time.

Longer term, the goal is to let someone scan a building once and give visitors a conversational guide they can use to explore it.

Built With

• ai
• ar