SAM Guide — Spatially Aware Mobile AI Companion

An always-on mobile AI that builds ongoing spatial awareness from camera and on-device sensors, speaks only when relevant, and adapts with context-aware reasoning and natural conversational restraint.

Comment

Inspiration

SAM Guide is inspired by a simple question: what if an AI assistant felt more like a loyal companion than a tool? The name SAM Guide is a deliberate reference to Samwise Gamgee — not a hero who takes over, but a constant, perceptive presence who supports, advises, and steps in only when it truly matters.

Most assistants today are either silent until prompted or overly chatty. We wanted to explore a middle ground: an AI that notices, understands context, and respects silence.

What it does

SAM Guide is an experimental Android app that continuously builds spatial and contextual awareness using the phone’s camera, motion sensors, and location data. It uses Gemini 3 to reason over live visual input, sensor state, and recent context to understand nearby objects, layout, and user movement.

It speaks only when there is a real reason—such as safety, relevance, or a direct user request—and adapts how often it initiates based on how the user responds.

How we built it

We built SAM Guide in Kotlin using Jetpack Compose and CameraX with Camera2 interop. Camera frames, camera intrinsics, and fused sensor data are sent to Gemini 3 Flash to enable multimodal spatial reasoning without relying on depth sensors.

A rolling context window with periodic state summarization keeps responses fast and coherent. A live developer HUD exposes sensor values and AI context for transparency during testing and demos.

Challenges we ran into

The hardest challenge was restraint. Making the AI talk less, not more, required careful prompt design and prioritization logic. Another challenge was enabling useful spatial reasoning with a single RGB camera while keeping latency acceptable.

Accomplishments that we're proud of

We’re proud that SAM Guide feels perceptive without being intrusive. In unedited, first-person tests, it correctly identifies objects, understands room layout, answers spatial questions, and stays silent when nothing needs to be said.

What we learned

Silence is a feature. Continuous multimodal context combined with strong behavioral boundaries can make AI feel more trustworthy than constant interaction.

What's next for SAM Guide — Spatially Aware Mobile AI Companion

Next, we plan to deepen long-term spatial memory, improve relative distance reasoning, and further personalize behavior—while keeping SAM quiet, loyal, and dependable.

Built With

Share this project:

Updates