GeminiAtlas

Intelligent AR assistant that answers your questions about the real world via Snap Lens and the Google Gemini API.

Comment

Inspiration

What if Pokémon GO was also a social experience that encourages your curiosity? A large part of today's short-form social media consists of being glued to your phone, browsing passively with constantly shifting attention – one of the main causes of the rise in learning difficulties worldwide. GeminiAtlas opposes every aspect of the "death scroll". It encourages you to go out and discover the world, to be a curious explorer, to connect with people across time and space, and to live and grow through your learning with Qwen Cloud.

What it does

GeminiAtlas is a Snap Spectacles filter that transforms your everyday environment into a personalized, social, AI-powered learning experience based on Qwen. It’s built on several pillars: First, I align the sky with you: the Constellations Lens. This is a Snap Lens in augmented reality that overlays constellations directly onto the real stars you see. With Lens Studio + Qwen Cloud, I integrated 3 key controls:

  1. Location: The user changes country or city. The sky recalculates in real time via the Qwen API.
  2. Compass: The Lens orients with you. You look North, it displays the Big Dipper.
  3. Voice control: The user says "Orion" or "Cassiopeia". Qwen-LLM calls up the constellation and highlights it. The result: The sky is no longer abstract. It becomes an interactive astronomy textbook. Second, I read for you: the Bookwoom Lens. Bookwoom is a Lens for Snap Spectacles. The principle is simple: the user frames a book with their hands. With Lens Studio, Qwen-VL analyzes the cover live. In 2 seconds, I pull the info via Qwen Cloud: author, summary, key themes, and similar books. The text leaves the cover and becomes a reading card you view with your eyes. The result: A library becomes a living database. Third, I show you the Globe. This is the heart of GeminiAtlas. Every card you capture with Constellations or Bookwoom has a position, a subject, a memory. Qwen Cloud organizes them spatially around you. You turn your head: Noguchi in Seattle. Another turn: innovation in Cotonou. Your long-term memory becomes a landscape. The more you record, the more the experience becomes your best partner. Capture and Remix! Capture any element of your environment with a simple two-hand pinch, and GeminiAtlas transforms it with Qwen-VL into a short, surprising Curiosity Card based on a random interest of your choice. You can then converse with the Qwen AI assistant about each card to get details and learn more. We’ve found time and again that the most memorable things we learn aren’t the ones we study in a structured way, but the ones we discover by chance.
  4. The intersection of two disciplines often leads to unexpected discoveries.
  5. Our interests are the best guides and windows for expanding our understanding and view of the world.
  6. Exploration and discovery naturally make you more curious and more of a learner; it’s a virtuous cycle that makes the world more exciting. Explore and Share! Cards are geolocated and designed to be shared via Qwen Cloud. Join your hands in front of your chest – a symbolic gesture that "uncovers the world" – and send a signal that analyzes your location and reveals a universe of cards left by you and others. Save, comment, follow, or share your own capture!
  7. When you find an interesting object, discover what others have learned about it – perhaps through an interest you never thought of before.
  8. When you visit a new place, explore it with those who came before you, sharing a moment across space and time. Relive and Remember — A living, evolving memory palace in the palm of your hand, where Qwen Cloud stores all your saved cards. The AI organizes everything spatially, by subject, relevance, and what it learns about you. The more you record, the better the experience understands how you learn. Collect and Battle! Take on the challenge and learn together by competing with your friends in a Q&A game. Questions are randomly generated by Qwen-LLM from both players’ cards; the one with the most cards and who knows them best wins. A Qwen AI host, with offbeat humor and audio-only, reads each question aloud and reacts to your answers.

How we built it

GeminiAtlas is a Spectacles lens built with Lens Studio. All our models are processed by Qwen Cloud. The assistant and battle host interact on Qwen-LLM; image understanding and battle question generation use Qwen-VL. From an architectural standpoint, the interface consists of many subsystems coordinated by a set of global.* singletons, allowing prefabs to reach a shared state without scene wiring. Each conversational agent connects to Qwen Cloud via Snap’s Remote Service Gateway RSG, streaming speech in 24 kHz PCM, with controlled transitions so only one live session is active at a time. The capture pipeline performs a two-hand crop to generate a Qwen-VL caption. The globe-to-card transition is handled by a guided state machine that mathematically aligns the sphere and the flat map by ground footprint. Battle mode is managed by the host on SpectaclesSyncKit, with questions generated from captured cards by Qwen-LLM and broadcast between the two players.

Challenges we ran into

Most of the difficulties were about coordination — between sessions, devices, and clocks with Qwen Cloud. The main issue was that RSG only keeps the most recent session active – two simultaneous Qwen sessions interrupt the connection – so we had to make our voices explicitly share a single time slot. The shared microphone is destructive and can lock up: a failed start right before launch leaves the provider "started but out of service", so we implemented a cyclic stop-restart mechanism and a monitoring system for recovery with Qwen Cloud. Multiplayer mode was a real challenge. Since device clocks aren’t comparable, the popularity leaderboard had to be based on the Qwen Cloud server clock. Also, Ulkit buttons misled us: their touch event handlers are only enabled at startup, so disabling a button at launch makes it permanently unusable.

Accomplishments that we're proud of

We’re proud that the assistant looks like a Qwen character, not just a chatbot: a strong personality, an expressive sphere reactive to sound, real-time text captions, all running via Qwen Cloud with no on-device intervention. The globe-to-card navigation is exactly as we imagined: one continuous gesture, from orbit to a street you can walk, with the transition invisible by design. Card discovery is truly magical: a plane wave in the room and cards appear as if by magic at every touch. And we deployed a real 2-player quiz, tolerant to latency, with an adaptive Qwen AI host that knows not to mock a player who’s already struggling.

What we learned

The common thread in everything we learned: decouple thinking from voice and make the deterministic parts boring. Our line banks, tool declarations, and query logic are completely independent of any model and easily testable; agents only own the live Qwen Cloud session. We learned to freeze what never changes and only generate what’s personal: the 37 fixed Cosmos cards instantly send pre-written questions for free, while only user-captured cards require a Qwen Cloud API call. We learned to maintain pure, deterministic placement calculations: every card scatter is seeded by the card ID, so markers never drift between frames, zooms, or sessions.

What's next for GeminiAtlas

More world, more curiosity with Qwen Cloud. Currently, the globe features Tokyo, Seattle, and Los Angeles with a universe of 37 cards and 13 initial interests. The next step is to expand the map and library to many more cities and themes with Qwen models, for a rich and varied discovery experience. We want to deepen the social dimension: more detailed profiles, the ability to follow people whose cards you like, and themed routes in a neighborhood. We’d like to develop Battle mode beyond two players, offering small-group games and themed decks generated by Qwen-LLM. Longer term, we want GeminiAtlas to become a living, shared atlas of human curiosity on Qwen Cloud: every wall, every dish, and every street corner annotated by a person who found it interesting, waiting for the next passerby. To conclude: GeminiAtlas + Snap Spectacles + Lens Studio + Qwen Cloud = learning by moving, sharing, playing. The buzzer is done waiting. Ready to start ?

Built With

Share this project:

Updates