Inspiration

Ideas rarely fit on a flat monitor. UluP Spaces turns projects into visual maps of connected nodes, and I wanted to step inside that map. UluP Spatial puts your whole project around you in mixed reality: nodes are spheres you grab, move and connect with your bare hands. Then I asked a harder question: what if you could simply talk to the map?

What it does

  • Grab, move and link nodes with hand tracking. Positions sync in real time to the same backend as UluP Spaces.
  • Ulupy, an assistant that lives in the map. Press the mic and ask out loud: "what's blocking this project?", "tell me about Launch", "summarize this project", "is it synced?". Ulupy reads your real project, answers in a small panel next to you and lights up the nodes it is talking about.
  • Ulupy builds with you. Say what you want ("add a website node with three tasks") and it shows a purple preview of the new nodes and tasks first. Say "Do it" and they appear in your space. Nothing is created until you confirm.
  • Bring your own AI. Connect any MCP-capable assistant to your project and what it creates shows up live in the headset, each node labelled "Ulupy".
  • Create nodes by voice: speak, watch the live transcript, pick a color, and a beam of light marks the new node. Speech recognition runs fully offline on the headset (English and Italian).
  • Try it with no account: Ulupy works on a demo project in one tap. If the connection drops, it falls back to a local summary of your project instead of failing.
  • Pin a node to your real room: the headset scans your space, you drop a node on a real surface like the door, and it stays there in passthrough. Switch to full VR and it rejoins the other nodes; switch back with a hand gesture and it returns to the door. Pinned spots are saved relative to your room scan.
  • Attach files to tasks: PDFs with zoom and page controls, and .glb 3D models you place, resize and spin in your room.
  • Set due dates from inside the headset, with a 3D Timeline that lines nodes up by date and a Focus view that uses depth to show what's blocking a project.
  • Work in passthrough mixed reality or a full immersive backdrop (Cosmos, Aurora, Space, Dusk, Void).
  • Start from community templates or a blank page, private by default. Guest mode takes one tap, and "Create account" upgrades the guest session in place so no project is lost.
  • Hands-first panels with large targets and hover feedback. They stay upright and follow you when you recenter, seated or standing, and the keyboard panel moves closer when you sit.
  • Offline where it counts: voice recognition runs entirely on the headset. Sync, accounts and Ulupy's AI answers need internet.
  • Room pinning needs a scanned room: run Space Setup on your Quest first (Settings > Physical space), then drop a node on the door in passthrough.

How I built it

I first shipped a WebXR version (v1.0.0, published on the Meta Horizon Store), then rebuilt it natively in Kotlin with the Meta Spatial SDK for the competition: ECS systems for grab, collisions, connections and spawn effects, Compose-based panels, the Mixed Reality Utility Kit for room scanning and pinning, Vosk for offline speech, Supabase for auth and data, and the UluP Spaces MCP server so your own AI can act inside the headset.

Ulupy runs on two layers. Simple questions (what's next, what's blocking, tell me about a node) are answered locally from the project data, so they are instant and work offline. Open questions go to a small AI model that gets a compact view of the project and must answer in a fixed format, with at most three nodes to highlight. Plans from the AI are checked before the preview appears, so a bad answer never turns into nodes in your space.

Challenges

Moving from WebXR to a native engine meant relearning input, panels and the scene model. I hit a Gradle/Kotlin build crisis and rebuilt the project from the official template. Opaque panels were needed to keep the hand pointer visible. Giving passthrough a real purpose meant tying each node to two places, one in your room and one in the VR universe, and switching between them cleanly.

Ulupy brought its own problems. Voice commands get misheard ("MCP" becomes all kinds of things), so I taught it the common mistakes. The Quest 3S has no text-to-speech, so Ulupy answers in text and with small sounds. The panel also had to stay in view without covering the nodes, and the microphone had to close cleanly when you dismiss it.

What's new in the competition window

UluP Spatial existed before as a WebXR prototype. Everything native listed above, including Ulupy and room pinning, was built after September 24, 2026.

What's next

Pinning to more surfaces (windows, walls, tables) and an unpin button, light mode, user-uploaded backdrops, day markers on the Timeline, pinned windows you can glance at without turning your head, spoken answers from Ulupy, and a public store launch planned after the competition (from the end of the year).

Built With

  • ai-assistant
  • android
  • hand-tracking
  • jetpack-compose
  • kotlin
  • mcp
  • meta-quest
  • meta-spatial-sdk
  • mixed-reality
  • mruk
  • nextjs
  • passthrough
  • postgresql
  • speech-recognition
  • supabase
  • vosk
Share this project:

Updates

posted an update —

UluP Spatial now has Ulupy. Ulupy is an assistant that lives inside your project map. Press the mic and ask what's blocking your project, what to do next, or for a summary. It reads your real project, answers in a panel and lights up the nodes it means. You can also tell it what to add. It shows a purple preview of the new nodes and tasks first, and nothing is created until you say "Do it". It works with the demo project in one tap, no account needed. The new build (version 6) is live in the Competition channel. Next up: more surfaces to pin nodes to, and spoken answers from Ulupy.

posted an update —

Room pinning, new trailer and a 60-second build update Update, October 25: UluP Spatial can now pin nodes to your real room.

The headset scans your space (Mixed Reality Utility Kit). In passthrough, drop a node on a real surface like the door and it stays there. Enter full VR and it rejoins the other nodes in the Universe; one hand gesture back to passthrough and it returns to the door. Run Space Setup on your Quest first.

Also new:

  • An updated trailer with the "Ask your AI" and room-pinning sections
  • A 60-second build update video: who I am, how I built it with Kotlin and the Meta Spatial SDK coming from WebXR, and what's next

Planned: pinning to windows, walls and tables, an unpin button, and a public store launch after the competition, from the end of the year.

Submission history