-
-
Omni-Reality
Omni-Reality: Your World Is Now the Interface
AI that doesn't just tell you. It shows you.
Inspiration
Artificial intelligence has become remarkably good at answering questions, but there is still a fundamental disconnect between what AI understands and how we experience the physical world.
Imagine asking an AI how to organize your workspace. It might give you a perfectly reasonable list of suggestions: move your cup away from your laptop, organize the cables, reposition your notebook. But you still have to interpret those instructions, identify the relevant objects, and translate the AI's words into physical actions.
Now imagine asking that same question and watching the answer appear directly in front of you.
Your cup lights up. The AI tells you where to move it. You say "next," and your notebook becomes highlighted. You look at an empty corner and start experimenting with furniture arrangements, placing, rotating, and resizing virtual objects in the same physical space you occupy.
That is the experience we set out to build at Hack the North 2026.
Omni-Reality is a conversational mixed-reality assistant that sees what you see, understands what you say, and brings its responses into your physical environment.
Rather than creating another chatbot with a camera, we wanted to explore a different relationship between humans and AI: one in which the world itself becomes the interface.
What We Built
We developed Omni-Reality as a native mixed-reality application for the Meta Quest 3S, combining multimodal AI, computer vision, spatial computing, and natural voice interaction.
The experience revolves around three complementary capabilities.
1. Talk to Your Surroundings
Omni can receive visual information from the Quest's passthrough cameras alongside the user's spoken requests, allowing it to reason about objects in the surrounding environment.
Ask it to identify an object or highlight something specific, and it can locate the requested object and overlay it with a colored segmentation mask in mixed reality.
The highlight follows the object across successive camera frames rather than remaining a static annotation on a photograph.
This combines conversational AI with visual grounding: the assistant can connect what the user says to what the user is actually looking at.
2. Guided Mode: Turn AI Advice Into Physical Instructions
Our Guided Mode transforms an ordinary conversation into a contextual, step-by-step visual tutorial.
For example, a user can look at their workspace and ask:
"Omni, show me how to organize this desk."
The multimodal AI analyzes a captured image of the desk alongside the spoken request and generates a structured plan involving relevant physical objects.
Omni might respond:
"First, move your mug to the right of your laptop."
Both objects become highlighted, with the mug emphasized as the primary target.
The user says:
"Okay, next."
The assistant advances to the next instruction, changes which objects are highlighted, and continues speaking.
The user can also say "repeat," "done," or "stop" to control the experience naturally.
Behind the scenes, our tutorial engine maintains the plan and its current step, while SAM 2 continues tracking the relevant objects. The user does not need to repeatedly identify or select the same objects.
The result is an assistant that can guide your attention through a real environment, one physical action at a time.
3. Layout Mode: Experiment With Your Space in Three Dimensions
We also built an interactive furniture-layout experience.
Using the Quest, users can place virtual furniture directly into their surroundings and experiment with different arrangements before moving or purchasing physical objects.
Our implementation includes a furniture catalog with sofas, armchairs, tables, bookshelves, and lamps.
Users can:
- Place multiple three-dimensional furniture objects in their environment.
- Move furniture around using the Quest controllers.
- Rotate objects to explore different orientations.
- Resize furniture interactively.
- View approximate dimensions and spacing between pieces.
- Walk around the arrangement and experience it from different perspectives.
This addresses a familiar problem: it is difficult to understand how furniture will occupy a room from photographs, measurements, or a conventional shopping website.
With Omni-Reality, you can experiment inside the actual space instead.
The furniture interaction is deliberately designed to remain responsive without requiring a cloud-model call for every manipulation. The headset handles the spatial interaction locally, using approximate dimensions rather than claiming precision room measurements.
Together, these capabilities let users move from understanding their surroundings to receiving guidance and experimenting with physical-space decisions.
How We Built It
Omni-Reality combines several existing technologies into one coordinated mixed-reality experience.
Our architecture separates the headset's latency-sensitive spatial interactions from the more computationally demanding AI and computer-vision components.
Meta Quest 3S + Unity
We built the headset application in Unity using C#, the Meta XR SDK, and Meta's Passthrough Camera API.
The Quest provides RGB camera frames, microphone input, camera geometry, and the spatial context needed to render virtual elements in the user's physical environment.
Huawei Qwen Omni + Multimodal Reasoning
Huawei's Qwen Omni models, accessed through the sponsored API gateway, provide the multimodal intelligence behind our visual grounding and guided interactions.
By processing captured images together with spoken requests, the model can understand references to visible objects, generate contextual responses, and produce structured instructions that our application can execute.
Rather than asking the model to generate Unity code or directly manipulate the headset, we constrain its output to validated object selections and structured actions.
SAM 2 + Object Tracking
We integrated Meta's Segment Anything 2 to transform AI-selected object coordinates into segmentation masks.
The multimodal model identifies what matters; SAM 2 determines which pixels belong to those objects and maintains their masks across subsequent frames.
Unity then renders the resulting visual highlights.
This division allows Omni to connect language understanding with precise visual feedback without repeatedly asking the language model to segment every frame.
Conversational AI + Speech
We also integrated Gemini Live for natural spoken conversation, alongside Huawei-powered voice and object-selection workflows.
The user can speak to the assistant, hear its responses, and request visual actions involving objects in their surroundings.
Python + WebSockets
A Python coordinator connects the Quest application, cloud AI services, and SAM 2 tracking pipeline.
WebSockets carry camera data, audio, tracking results, and spatial instructions between the components.
The coordinator also manages tutorial state, associates tracked objects with their identities, synchronizes spoken instructions with visual highlights, and handles transitions between interactions.
The overall architecture is:
Quest camera + microphone → Multimodal AI → Structured reasoning and object selection → SAM 2 → Unity spatial overlays + spoken guidance
Our separate Layout Mode adds an interactive 3D spatial-manipulation layer directly on the headset.
Challenges We Faced
Our biggest challenge was not implementing any individual feature. It was making technologies with very different responsibilities behave like one coherent system.
Connecting AI reasoning to physical space
A language model can identify a cup in an image, but that does not automatically tell a headset where to display a highlight as the user moves.
We had to connect image coordinates, camera geometry, frame identities, object segmentation, and Unity rendering while accounting for the delay between capturing an image and receiving the model's response.
Making object tracking responsive
Running computer vision on a continuous camera stream introduces significant computational and memory constraints.
We had to balance tracking quality, frame processing, model inference, and the limited GPU resources available during the hackathon.
We also needed to prevent delayed results or changing object selections from producing incorrect visual feedback.
Synchronizing speech with visual guidance
Guided Mode introduced another challenge: the assistant's spoken instruction and the highlighted object must represent the same tutorial step.
We developed a stateful tutorial system that maintains object identities and progresses through instructions without regenerating the entire plan or restarting the tracker whenever the user says "next."
Making 3D interactions feel natural
Creating movable furniture was only the beginning.
We had to implement placement, selection, rotation, resizing, approximate dimensions, and spatial feedback in a way that worked through the Quest's controllers.
We also discovered that model-generated arrangements and cloud speech could introduce unnecessary delays into interactions that should feel immediate. Keeping frequent furniture manipulations local made the experience more responsive and reduced its dependence on external APIs.
Bringing everything together under hackathon constraints
Finally, we had to integrate multiple development environments, hardware devices, cloud APIs, and open-source projects within a 36-hour hackathon.
The project required us to make difficult trade-offs between ambitious capabilities, performance, reliability, and the experience we could actually demonstrate.
What We Learned
The most important lesson was that multimodal AI becomes substantially more useful when its outputs can affect how a person interacts with their environment.
Vision gives the assistant context. Audio makes interaction natural when the user's hands are occupied. Language enables reasoning and explanation. Computer vision connects that reasoning to specific physical objects. Spatial computing makes the result visible in the world.
None of these capabilities alone delivers the same experience.
We also learned that building a convincing mixed-reality AI application is fundamentally a systems-integration problem. A sophisticated model is only one part of the solution. Timing, object identity, camera geometry, state management, rendering, and user feedback are equally important.
Perhaps most importantly, we learned when not to use AI. Rendering and manipulating furniture should not require an expensive model call every time someone moves their hand. We used AI where contextual understanding adds value and conventional computation where immediate, predictable behavior matters.
What's Next
We see Omni-Reality as a foundation for a new category of spatial AI assistants.
The same interaction model could eventually support equipment tutorials, interactive education, household organization, furniture planning, accessibility, and hands-free guidance in unfamiliar environments.
Our next steps include deeper integration between conversational reasoning and 3D spatial manipulation, richer visual instructions, and more persistent understanding of the user's surroundings.
We believe the future of AI interaction extends beyond text boxes, mobile screens, and spoken answers.
Instead of asking people to adapt to an AI's interface, Omni-Reality explores what happens when AI adapts to the world people already inhabit.
Don't just ask AI. See the answer.
Built at Hack the North 2026.
Log in or sign up for Devpost to join the conversation.