Walk Me There

01-hero
02-problem
03-loop
04-architecture
05-hardrule
06-onroute
07-wrong direction
08-teach moment
09-memory
10-firestore-user
11-firestore-episode
12-cloudrun
13-reproducibility
14-hardware
15-ending

Inspiration

Maps are remarkably good at knowing where the route is. But knowing the route and successfully following it are not the same problem.

Walk Me There started from a simple frustration: even with a correct route on screen, people can still walk the wrong way, misread orientation, hesitate at an intersection, or fail to translate abstract instructions like “head east” into a physical action.

Most navigation products treat this as a display problem: show the route more clearly.

We wanted to treat it as an agent problem instead.

What if navigation could notice whether the human is actually succeeding, understand how that individual interprets directions, adapt its guidance, and stay involved until recovery is confirmed?

That became the core thesis:

Maps know the route. Walk Me There helps the human actually follow it.

What it does

Walk Me There is a stateful walking companion built around a continuous loop:

Observe → Decide → Guide → Verify → Recover

A deterministic navigation engine observes movement and owns geographic truth: whether the user is on route, walking the wrong way, or off route.

When recovery is needed, Gemini turns those validated facts into guidance this particular person can understand.

The important difference is that success is not defined as “the agent produced an answer.” Success means the person actually gets back on the correct path.

And Gemini is not allowed to grade itself. It can guide the recovery, but only the deterministic navigation engine can verify subsequent movement and certify that recovery succeeded.

Walk Me There also remembers how an individual understands orientation.

For example, when a user says that east/west language is not useful, that preference is persisted in Firestore and future guidance avoids cardinal directions. When the user later says clock directions work better, the profile can store:

orientationVocab = "clock"

That memory does not simply sit in a database. It changes how the owl guides the user next time.

The profile also keeps free-text notes and an evidence trail quoting the user's own words, so every stored preference is traceable to what the person actually said.

The live application can also speak validated guidance using Gemini TTS. Voice sits strictly downstream of the deterministic text gates: it reads the approved on-screen words verbatim and makes no navigation decisions of its own.

How we built it

The key architectural decision was to separate geographic truth from human guidance.

The deterministic navigation engine computes navigation state from raw observations. The server recomputes that state instead of trusting a client-declared result.

Only after the engine confirms that recovery is needed does Gemini receive the validated navigation facts plus the user’s persistent profile. Gemini can then decide how to express the next useful action and can update the user model when the person teaches it something about how they understand directions.

Before any generated recovery guidance reaches the screen or voice layer, it passes through a deterministic validator.

The production stack uses:

Gemini 3.5 Flash on Vertex AI for adaptive guidance and user-model updates
Google GenAI SDK (@google/genai)
Cloud Run for the deployed application and companion service
Firestore for persistent user profiles and recovery episodes
Gemini TTS (Puck) as a presentation-only voice layer
A deterministic navigation engine and validator for geographic truth, verification, and safety boundaries

Recovery episodes are opened when the engine confirms that intervention is needed and closed only after the engine observes a stable return to the route.

This gives Walk Me There a real closed loop rather than a one-shot prompt-response interaction.

Challenges we ran into

The hardest problem was deciding what the model should not be allowed to decide.

A language model can generate convincing navigation language very easily. But convincing language is not geographic truth when someone is physically moving through the world.

So we drew a hard boundary.

The navigation engine decides where the user is and whether recovery succeeded. Gemini decides how to communicate within those facts. A deterministic validator sits between the model and recovery guidance shown to the user.

A second challenge was memory.

Saving “this user prefers clock directions” is easy. Making that memory reliably change future guidance — including across reloads — required persistent user state rather than temporary conversational context.

A third challenge was reproducible testing.

Real walking behavior is difficult to reproduce exactly, so ?sim=1 drives the application with a scripted GPS trace covering on-route movement, wrong direction, recovery, off-route movement, and recovery again.

The movement is scripted; the execution is real. Cloud Run, Gemini, Firestore, server-side state recomputation, validation, and memory writes all execute live.

Accomplishments that we're proud of

We are proud that Walk Me There does not ask the language model to do everything.

The working system detects navigation states deterministically, wakes Gemini only when appropriate, generates user-specific corrective guidance, accepts feedback about how directions should be expressed, persists those preferences in Firestore, applies them to future guidance, and verifies recovery through subsequent movement.

The model cannot certify its own success.

The server does not trust client-declared navigation state.

The voice layer cannot rewrite approved guidance.

And if Gemini or TTS fails, navigation still falls back safely to deterministic text.

The current production build is covered by 60 automated tests: the original 31-test navigation and validator baseline plus 29 tests added with the runtime voice layer.

Most importantly:

We changed the unit of success from “gave directions” to “the human successfully followed them.”

What we learned

We learned that memory becomes much more valuable when it changes behavior rather than merely recalling information.

For navigation, knowing that someone understands “9 o’clock” better than “west” is not cosmetic personalization. It can determine whether the instruction is usable at all.

We also learned that real-world agents need verification loops.

When an agent acts on something outside the screen, producing an instruction is only the midpoint of the job. The system still needs to observe what happened and decide whether intervention is still necessary.

And we learned that being more agentic does not mean giving the model more authority.

For Walk Me There, the stronger architecture came from giving Gemini the part it is good at — understanding the person and communicating clearly — while keeping geographic truth and success verification deterministic.

An agent operating in the real world should reason about outcomes, not just outputs.

What's next for Walk Me There

The next step is field calibration of the navigation engine on real walking behavior, especially heading stability when a person turns around.

After that, we want to expand beyond the current hardcoded test route with real route and landmark data, while preserving the same architecture: deterministic geographic truth, adaptive human guidance, and verified recovery.

The user model can also grow over time to capture additional ways people understand navigation — landmark-based guidance, instruction timing, preferred level of detail, or when reassurance is more useful than a direct correction.

Longer term, we imagine Walk Me There moving beyond the phone.

Our future product concept is a small physical owl companion that communicates through light, vibration, and voice, allowing guidance to become more ambient while the person keeps their eyes on the world instead of a map.

The body may change.

The goal does not:

Help the human actually get there.