GeminiAtlas

• 

  GeminiAtlas

Inspiration

What if Pokemon GO was also a social experience that encourages your curiosity? A large part of today's short-form social media consists of being glued to your phone, passively scrolling, with constantly shifting attention — one of the main causes of rising learning difficulties worldwide. GeminiAtlas opposes every aspect of the "death scroll." It encourages you to go out and discover the world, to be a curious explorer, to create connections with people across time and space, and to live and grow through learning.

What it does

GeminiAtlas is a Snap Spectacles filter that transforms your everyday environment into a personalized, social, AI-powered learning experience. It is built on several pillars: First, I align the sky with you: Lens Constellations. This is an augmented reality Snap Lens that overlays constellations directly onto the real stars you see. With Lens Studio, I integrated 3 key controls:

1. Location: The user changes country or city. The sky recalculates in real time.
2. Compass: The Lens orients with you. You look North, and it displays the Big Dipper.
3. Voice control: The user says "Orion" or "Cassiopeia." The Lens calls the constellation and highlights it. The result: The sky is no longer abstract. It becomes an interactive astronomy textbook. Second, I read for you: Lens Bookwoom. Bookwoom is a Lens for Snap Spectacles. The principle is simple: the user frames a book with their hands. With Lens Studio, GeminiAtlas analyzes the cover live using the Gemini 2.5 Flash multimodal API. In 2 seconds, it surfaces the info: author, summary, key themes, and similar books. The text leaves the cover and becomes a reading card you view with your eyes. The result: A library becomes a living database. Third, I show you the Globe. This is the heart of GeminiAtlas. Every card you capture with Constellations or Bookwoom has a position, a topic, and a memory. I organize them spatially around you. You turn your head: Noguchi in Seattle. Another turn: innovation in Cotonou. Your long-term memory becomes a landscape. The more you capture, the more the experience becomes your best partner. Capture and Remix! Capture any element in your environment with a simple two-hand pinch, and GeminiAtlas transforms it into a short, surprising Curiosity Card using Gemini 2.5 Flash, based on a random interest of your choice. You can then engage in conversation with the Gemini Live assistant about each card to get details and learn more. Explore and Share! Cards are geolocated and designed to be shared. Bring your hands together in front of your chest — a symbolic gesture that "uncovers the world" — and it sends a signal that analyzes your position and reveals a universe of cards left by you and others. Save, comment, follow, or share your own capture! Relive and Remember — A living, evolving memory palace in the palm of your hand, where all your saved cards are stored. Gemini organizes everything spatially, by topic, relevance, and based on what it remembers about you via Vertex AI Embeddings. Collect and Battle! Take on the challenge and learn together by competing against friends in a Q&A game. Questions are generated randomly from both players' cards by Gemini 2.5 Flash; the one with the most cards and who knows them best wins. A Gemini Live AI host, with quirky audio-only humor, reads each question aloud and reacts to your answers.

How we built it

GeminiAtlas is a Spectacles Lens built with Lens Studio. All our models are processed by Snap's Remote Service Gateway (RSG). Google Cloud Stack Used:

1. Gemini 2.5 Flash via RSG: Primary multimodal model for vision, text, and reasoning. Powers Bookwoom, Capture, and Battle Q&A.
2. Gemini Live API via RSG: Real-time voice agent for the assistant and the Battle host. Streams 24 kHz PCM audio with low latency.
3. Vertex AI Embeddings: Used to index all captured cards for semantic search in the Globe and Relive memory. From an architectural standpoint, the interface consists of many subsystems coordinated by a set of global.* singletons. Each conversational agent connects to Gemini Live via RSG, with controlled transitions so only one live session is active at a time. The capture pipeline performs a two-hand crop and sends the image to Gemini 2.5 Flash for captioning and card generation. The transition from globe to map is handled by a guided state machine. Battle mode is managed by the host on SpectaclesSyncKit, with questions generated by Gemini 2.5 Flash and streamed between the two players.

Challenges we ran into

Most of the difficulties concerned coordination — between sessions, devices, and clocks. The main issue was that RSG only keeps the most recent session active — two simultaneous Gemini Live sessions interrupt the connection — which forced us to make our voices explicitly share a single time slot. The shared microphone is destructive and can lock up: a failed start leaves the provider "started but out of service," so we implemented a cyclic stop-restart mechanism and a watchdog system for recovery. Multiplayer mode was a challenge. Since device clocks are not comparable, the leaderboard is based on the Google Cloud server clock. Also, UIKit buttons misled us: their touch event handlers only activate on startup.

Accomplishments that we're proud of

We are proud that the assistant has the appearance of a character: a strong personality, an expressive sphere reactive to sound, and real-time text captions, all running via Gemini Live on RSG with no on-device model. The navigation from globe to map is exactly as we imagined: a single continuous gesture, from orbit to a walkable street. Card discovery is truly magical: a wave across the room and cards appear as if by magic at every touch. And we deployed a real two-player quiz, tolerant to latency, with an adaptive Gemini Live host.

What we learned

The common thread: decouple thinking from voice and make deterministic parts boring. Our line banks, tool declarations, and query logic are independent of any model and easily testable; agents only own the Gemini Live session. We learned to freeze what never changes and only generate what is personal: the 37 fixed Cosmos cards send pre-written questions, while only user-captured cards require a Gemini 2.5 Flash API call. We learned to maintain pure, deterministic placement calculations: each card scatter is seeded by the card ID, so markers never drift between frames, zooms, or sessions.

What's next for GeminiAtlas

More people, more curiosity. Currently, the globe offers Tokyo, Seattle, and Los Angeles with a universe of 37 cards and 13 initial interests. Next: expand with Vertex AI Vector Search for millions of cards. We want to deepen the social dimension with Google Cloud Firestore for profiles and follow systems. We will expand Battle mode beyond two players using Gemini Live multi-agent orchestration. Long term: GeminiAtlas becomes a living, shared atlas of human curiosity, fully powered by Google Cloud.

To conclude: GeminiAtlas + Snap Spectacles + Lens Studio + Google Cloud = learning by moving, sharing, and playing. The buzzer is done waiting. Ready to begin?

Built With

• 24-khz-pcm-streaming-development:-typescript/javascript
• clock
• edge
• functions
• gemini
• glsl-shaders
• gpt-4o
• handwritten
• openai-gpt-4o-vision-api
• platform-&-engine:-snap-lens-studio
• server
• snap-cloud-storage-for-card-geolocation-multiplayer:-spectaclessynckit-for-real-time-2-player-sync
• snap-spectacles
• spectaclessynckit-ai-&-apis:-google-gemini-live-api-via-snap-remote-service-gateway-(rsg)
• state-machine-for-globe-to-map-transition
• supabase
• ulkit-ui-architecture:-singleton-pattern-global.*
• voice