Inspiration
When I first began architecting this system, Gemini Live had not yet been integrated with Google Maps. I was cycling the trails of Fort Mill, SC, caught in a sudden storm, and realized a fundamental possibility to leverage no context-switch of eyes and hand coordination by including auditory navigation through live copilot for not just mobile navigation, but to answer specific user queries on the go, eyes and mind on the road for sustained focus: standard apps are built to dictate directions only, no conversational guide support.
By the time development was underway, Gemini-Maps integration rolled out, providing a conversational baseline. However, due sandboxing of apps in memory direct transfer of current user latitude longitude is not conveyed to Gemini and no such permission toggle for user to accept exists. There Gemini cannot directly access user geo data with current movement despite this integration. While Gemini Live is an incredible speech-to-speech interface, it could be leveraged for high-resolution "eyes" on your immediate, off-grid environment. As humans, we don't just want to be told to "turn left"; we have instant, subjective questions on the road that standard navigation isn't configured to answer—questions like
- "Is there a smoother path on the other side of this trail?"
- "What's the surface condition ahead—can I maintain speed?"
- "Where's the nearest water fountain from here?"
Everyone understands their surroundings differently. While riding, driving, or even walking, instant queries emerge that require you to describe what you're seeing and receive a conversational response you can hear through your ears. You can't pull out your phone. You can't type. You need to speak and listen—but the AI responding to you needs accurate, real-time geospatial awareness of your exact position and movement.
This became even more urgent when I watched my grandparents struggle on daily walks. Standard GPS apps "snap" them to the nearest road, completely ignoring the uneven pavement, steep curbs, or 1-meter potholes that create real fall risks. They needed a voice-first co-pilot that understood their precise location at a granular level—not just "you're on Main Street," but "uneven pavement 2 meters ahead, handrail available to your left."
Current AI assistants aren't "spatially accurately aware." Gemini Live offers incredible conversational AI, and with high-frequency location tracking, IMU fusion, and sub-meter precision (S2 Level 24 ≈ 58cm cells), it can also serve as a true hands-free geospatial companion.
The S2S Geospatial Adapter bridges this gap - through purposeful geo tracking, transforming Gemini Live into a specialized co-pilot that knows where you are, what's around you, and how you're moving at every moment, answering your questions conversationally while you stay focused on the path ahead.
What it does
The S2S Geospatial Adapter is a high-performance C++ engine that transforms Gemini Live into a specialized, real-time safety co-pilot. It focuses on the technical "blind spots" of standard navigation: Utilized OS-level Interprocess Communication (IPC) to optimize real-time data streams to avoid RPC calls for geo data available in registers of different application sandboxes.
- Accessibility & Senior Safety: Provides stability alerts like "Uneven pavement 2 meters ahead" or "Handrail available to your left," detected via sensor fusion.
- High-Precision Tracking: Fuses GPS with IMU (Inertial Measurement Unit) data using Kalman Filters and Pedestrian Dead Reckoning (PDR) for tracking at S2 Level 24 (~58cm) precision.
- Surface Quality Detection: Monitors the IMU to detect real-time shifts (e.g., asphalt to gravel) and alerts the user before they lose traction.
- "Thinking While Moving": Eliminates the "deliberation bottleneck" by initiating spatial lookups for upcoming decision nodes while the user is still in motion, delivering answers in <500ms.
- Modular Plugin Engine: A domain-agnostic core that can swap between a
CyclingPlugin, anAccessibilityPlugin, or aSocialDiscoveryPluginvia a unified provider interface.
Gemini Tools Used
- Gemini Multimodal Live API: The backbone of the bidirectional, low-latency speech-to-speech session.
- Gemini 2.0 Flash: Selected for its exceptional speed and efficiency in processing high-frequency spatial metadata.
- Google AI Studio: Used for rapid prototyping of the "Geospatial System Instructions" and fine-tuning the agent’s descriptive style.
- Agent Development Kit (ADK): Leveraged for orchestrating the modular plugin architecture and managing real-time tool calling.
How we built it
We architected the system using C++23 to achieve the sub-millisecond latency required for safety-critical movement.
- Location Daemon: A persistent C++ service that derives the "true" state from noisy sensors using a Kalman Filter: $$\hat{x}{k|k} = \hat{x}{k|k-1} + K_k(z_k - H\hat{x}_{k|k-1})$$
- S2 Spatial Indexing: Uses the Google S2 Geometry library to hash coordinates into 64-bit cell IDs for rapid neighbor lookups and metadata retrieval.
- Zero-Copy Shared Memory IPC: Implemented Boost.Interprocess (Ring Buffer) to communicate between the daemon and adapter, keeping internal transmission latency below 1 microsecond.
- WebSocket Integration: Streams 16kHz audio while dynamically updating the agent's context instructions based on the user's active S2 cell.
Challenges we ran into
- The Rollout Pivot: When Gemini integrated with Maps globally, we pivoted from building a "Map App" to building a specialized Geospatial Context Engine for high-frequency sensor fusion and off-road cases.
- Elderly-First UX: Calibrating "Barge-in" logic to ensure safety alerts for my grandparents were timely enough to prevent a fall without being intrusive or startling.
- Concurrency: Synchronizing asynchronous GPS telemetry with steady PCM audio buffering required a lock-free Single-Producer Multi-Consumer (SPMC) queue.
Accomplishments that we're proud of
- Achieving sub-millisecond internal state updates, allowing the AI to "feel" the terrain with the user.
- Successfully tracking movement as granular as a single step or a curb descent.
- Building a system that serves both high-speed athletes and seniors with limited mobility.
What we learned
- The Noise Axiom: Raw sensor data is noisy ("False"); rigorous mathematical modeling is mandatory before an LLM can provide reliable safety utility.
- Persistence Layers: Separating the Location Daemon from the AI Session is the only way to maintain tracking if a voice call drops.
What's next
- Indoor Mapping: Integrating with Google Home for hands-free, room-level household assistance.
- Hardware Porting: Deploying the C++ daemon to run on smart glasses and low-power embedded bike computers.
- Offline SLMs: Integrating local Small Language Models to handle navigation when cellular data is lost in remote forest trails.
Log in or sign up for Devpost to join the conversation.