SpatialYolo

Visual Intelligence building on XR Devices (like Apple Vision Pro) Talking to Gemini in Live and ask OpenClaw to add item in front of people to cart online

Comment

SpatialYOLO Live

SpatialYOLO Live is a spatial AI agent for Apple Vision Pro. It is designed to let users interact with AI through live visuals, voice, and spatial context instead of typing everything into a text box. And it can ask OpenClaw to add item in front of people to cart online

Inspiration

The project was inspired by a simple observation: most AI assistants still live inside a text box, while real-world interaction happens in 3D space. We wanted to build an assistant that can truly see, hear, and respond in real time, so users do not need to stop and describe their surroundings manually. Instead, they can interact naturally with AI through gaze, live visuals, and voice.

How We Built It

We built the project with Swift, SwiftUI, RealityKit, ARKit, Vision, and CoreML, using the Gemini Live API as the core of the real-time multimodal interaction.

On-device, the app performs object detection and estimates spatial position and distance, then overlays that understanding as a spatial HUD inside the immersive Vision Pro experience. At the same time, sampled video frames and microphone audio are streamed through the Gemini Live API, allowing the model to respond instantly with both speech and subtitles.

To improve accuracy, we do not rely on raw video alone. We first extract structured scene context locally, such as object classes, relative position, distance, and facial-expression cues, and send that grounded context together with the live video stream to Gemini. We also added automatic scene narration, session recording, and memory playback, so the product feels more like a persistent spatial companion than a simple chat interface.

What We Learned

One of our biggest lessons was that the hard part of building a real-time multimodal agent is not just connecting a model. The real challenge is orchestrating the full experience.

With the Gemini Live API, we were able to unify vision, voice, and live response in one interaction loop, but we also learned that latency, concurrency, interruption handling, grounding, and spatial interface design all need to work together for the experience to feel natural and trustworthy.

We also found that combining local deterministic perception with a live multimodal model makes the system more stable, less ambiguous, and much better suited for spatial computing.

Challenges We Faced

The biggest challenges were both technical and experiential.

On the technical side, we had to deal with Vision Pro camera permissions, on-device inference performance, frame compression and transmission, real-time audio capture and playback, and session management issues such as connection stability and response timing with the Gemini Live API.

On the experience side, we needed to present useful information without blocking the real world, and make the agent feel responsive while still grounded in spatial context. This project is our exploration of what the next generation of AI interaction can look like: when AI is no longer trapped inside a text box, but becomes part of the space around the user.

Built With

Share this project:

Updates

Submission history