TalkTalkXR

An MR learning companion that helps you understand whatever you’re looking at and creates a mindmap to ground your knowledge.

Inspiration

Most of the time, when we see something we don’t recognize, our instinct is to “quick search” it on a phone or laptop. That works, but it’s cramped and linear: one screen, one page, one scroll at a time.

We built TalkTalkXR because mixed reality gives us something better: space.

In XR, you’re not limited to a single flat page. You can spread concepts out, pin ideas around you, walk through them, and navigate a mindmap as if it were a physical object. TalkTalkXR takes that everyday moment of curiosity—“what is this, and how does it work?”—and turns it into a spatial learning experience you can explore, rearrange, and revisit, directly in the world around you. TalkTalkXR is our attempt to turn casual curiosity into an ongoing learning habit.

What it does

TalkTalkXR turns your field of view into an interactive knowledge graph you can explore in space.

• Instant analysis
  With a simple gesture, you capture what you’re seeing. The app identifies key objects, infers what you’re likely interested in, and produces a structured breakdown of the scene.

• Learning Mindmaps
  Instead of a long paragraph, results appear as an interactive Mindmap with branches like:

  • “Identify” – what you’re looking at
    
  • “How it works” – mechanisms, structure, or function
    
  • “Related concepts” – nearby ideas worth knowing
    
  • “Learning path” – how to go deeper from here
    

• Spatial Library
  You can save any Mindmap into a persistent Spatial Library, gradually building your own knowledge collection anchored in the things you’ve actually seen.

• Integrated browser
  A built-in spatial browser lets you open articles, docs, or videos linked from the Mindmap without leaving MR, so deep dives live right next to the original context.

• Microgesture control
  Subtle finger movements handle scrolling, navigation, and panel management, keeping interactions light and fluid instead of relying on big arm gestures or controllers.

How we built it

• Core stack
  The app is built natively for Android in Kotlin, on top of the Meta Spatial SDK.

• UI/UX
  The interface uses Jetpack Compose with a custom MR-friendly design system: high contrast, generous spacing, and elevated cards tuned for readability in a headset.

• AI integration
  We use Gemini 2.5 Flash via the Gemini API. A strict system prompt and schema push the model to return structured JSON, which we feed directly into our Mindmap renderer.

• Gesture system
  A custom gesture router listens to Meta’s spatial input events and maps specific microgestures (like a thumb swipe) to synthesized Android touch events, enabling natural scrolling, back/forward navigation, and tab switching in standard views.

• Persistence
  A repository layer serializes Mindmap objects to local storage and restores them on app launch, so your library and past sessions survive restarts and feel consistent.

Challenges we ran into

• Taming the LLM
  Getting clean, parseable JSON from a creative model turned out to be one of the hardest parts. Smart quotes, stray commas, and unescaped characters would regularly break parsing. We built a JSON “sanitizer” that uses regex-based repair passes to fix common issues before we hand data to the parser.

• Spatial vs. content scrolling
  In MR, “move this window in space” and “scroll the text inside it” can feel very similar. Early on, we routed gestures incorrectly. We ended up refining our interpreter so depth-oriented “grab” gestures move panels, while focused swipes within content zones are treated as scrolling.

• State reconstruction
  The Library feature required us to dehydrate a rich Mindmap state into flat JSON and later rebuild it exactly. To make this reliable, we reworked our ViewModel so the UI always derives from a single serialized Mindmap representation, making load/restore deterministic.

• UI Rendering
  We encountered UI glitches and UI mis-rendering when we applied specific layout parameters from 2D to 3D space. We had to adjust the layout system to better handle the unique challenges of spatial UI.

Accomplishments we’re proud of

• A strong “magic moment”
  The jump from a raw image to a clean, clickable Mindmap feels immediate and intuitive. It turns the world around you into something you can browse and drill into.

• Resilient parsing pipeline
  Our “fail-soft” JSON handling significantly reduces crashes and hard errors. Even when the model output is slightly malformed, the sanitizer usually recovers it and keeps the experience smooth.

• Natural microgestures
  Scrolling a long article or navigating history with tiny thumb motions while panels float in front of you feels surprisingly native—more like manipulating content than using a traditional app.

What we learned

• Prompting isn’t enough on its own
  Simply asking for JSON doesn’t cut it. You need strict examples, a clear schema, and a validation/repair layer if you intend to treat model output as structured data in production.

• Spatial UI has a cognitive cost
  Every extra floating element adds mental overhead. Moving from a crowded dashboard to a leaner tabbed layout improved clarity and kept attention on the content, not the chrome.

What’s next for TalkTalkXR

• Voice interaction
  Add full voice support so you can just ask, “What’s that component?” or “Explain this setup” while looking at it—no menus or typing required.

• Shared spatial learning
  Enable multiple people to see the same annotations and Mindmaps anchored to the same real-world objects, unlocking teacher–student, lab, and guided-tour scenarios.

• Continuous video analysis
  Evolve from single-frame snapshots to real-time understanding, so explanations and labels can update dynamically as you move around and change your viewpoint.

• Real World Analysis
  Real-time analysis of what users see and guide them through mindmap, such as cooking, furniture assembly and other real-world tasks.

Built With

• gemini
• kotlin
• spatialsdk