Inspiration
Spatial computing and generative AI have opened up exciting possibilities, but most XR experiences still depend heavily on controllers, hand tracking, and repeated gestures. For people with limited mobility, motor difficulties, or physical fatigue, these interactions can be difficult or exhausting.
At the same time, most AI tools are still limited to chat boxes, where users type prompts and wait for results. We wanted to create something different, where AI becomes an active part of the 3D environment.
That's why we built WeaVR (Weave Anything), a voice-first, AI-powered spatial workspace for Meta Quest.
Our idea was to bring together two important concepts: Accessibility-First XR Design and Agentic AI Interaction.
We wanted to create an experience where users don't need to rely on hand gestures or controllers. Instead, they can use natural voice commands, line-of-sight awareness, and an intelligent spatial agent to create, organize, and interact with their workspace.
Our goal is simple: Make XR creation accessible to everyone, with AI doing the complicated work.
What it does
WeaVR is an accessible, voice-first creative workspace where users can generate images, videos, 3D models, interactive charts, and complete AI workflows inside a virtual environment.
At the center of WeaVR is an intelligent spatial AI agent that doesn't just answer questions. It understands the workspace, makes decisions, and performs actions for the user.
Accessibility-First XR Experience
Voice as an alternative to hand interactions: Users can create, move, resize, and connect objects simply by speaking. No controller, pinching, or physical grabbing is required.
Eyes-only and line-of-sight awareness: The AI understands what the user is facing, allowing them to ask questions about nearby objects without manually pointing at them.
Hands-free navigation and workspace control: Commands like "Move the video to the left," "Bring the chart closer," or "Make the model bigger" let users control their environment without reaching for objects.
High-contrast and readable interface: We designed nodes with distinct colors, clear connections, and readable text to make the workspace easier to understand.
Comfortable spatial layout: Objects automatically appear within a comfortable viewing area, reducing unnecessary head movement and physical fatigue.
Accessible task management: Users can create checklists, ask about pending tasks, and mark items as completed entirely through voice commands.
Agentic AI Interaction
Autonomous spatial agent: Instead of manually performing every action, users can describe their goals, and the AI decides which tools and steps are needed.
AI-powered workflow creation: For example, saying "Connect my kingfisher image to the video generator" makes the agent identify the correct nodes, check their connections, and link them automatically.
Multimodal content generation: Users can generate images using Runway, 3D models using Meshy, and videos using Runway, all from the same workspace.
Procedural 3D visualizations: The AI can create interactive 3D pie charts, bar charts, and mathematical models directly in the environment.
Context-aware assistance: The agent understands the active nodes, their content, and the user's viewing direction, allowing it to respond based on what's happening in the workspace.
Real-time voice interaction: Powered by ElevenLabs Conversational AI, the agent can listen, respond, and perform spatial actions while maintaining a natural conversation.
For example, a user can say:
"Create a 3D bar chart of the fastest birds, generate an image of a kingfisher, turn it into a video, and add these to my task checklist."
The agent can use different tools, create the required nodes, organize the workspace, and update the checklist.
WeaVR turns AI from a simple chatbot into an active creative partner inside XR.
How we built it
We built WeaVR using Meta's Immersive Web SDK (IWSDK) and Three.js to create a spatial environment that runs on Meta Quest.
For the interface, we used @drawcall/uikitml and designed a node-based workspace with high-contrast colors, visible connections, and comfortable positioning.
The main part of WeaVR is its ElevenLabs Conversational AI agent. We connected the agent to the workspace using real-time WebSockets and custom tools.
These tools allow the agent to perform actions such as:
- Creating and connecting AI workflow nodes.
- Moving, resizing, and organizing spatial objects.
- Generating images, videos, and 3D models.
- Creating interactive 3D charts and mathematical visualizations.
- Managing checklists and tracking completed tasks.
- Understanding active nodes and inspecting the user's surroundings.
We also built a context-aware spatial vision system using WebXR camera information. This helps the AI understand what the user is facing instead of analyzing the entire environment.
For content generation, we integrated Runway for images and videos and Meshy for 3D models.
We used Three.js to generate procedural 3D charts and mathematical shapes, allowing the AI to change colors, highlight elements, and explain concepts in real time.
A Node.js backend handles AI requests and video streaming, helping generated content appear smoothly inside the VR workspace.
Most importantly, we designed the entire experience around voice-first interaction and accessibility by default, rather than adding accessibility as an extra feature.
Challenges we ran into
1. Replacing hand gestures with voice commands: Making the AI understand commands like "Move the second image" or "Connect the duck image to the video node" without requiring users to point at objects. We built a system that identifies nodes using their type, position, content, and context.
2. Making XR comfortable: Constant head movements can be tiring, especially for users with limited mobility. We designed a curved workspace that keeps objects within a comfortable viewing area and allows users to reposition them through voice commands.
3. Context-aware spatial vision: The AI initially struggled to identify objects correctly when analyzing the entire environment. We improved this by focusing on what the user is directly facing.
4. Real-time agent interactions: We combined local command recognition with AI tool calls to make voice-controlled actions faster and more natural.
Accomplishments that we're proud of
- Accessibility by Default: Built a hands-free XR workspace where users can create, move, and connect objects entirely through voice, reducing the need for controllers and gestures.
- Agentic AI Interaction: Created an intelligent spatial agent that understands the workspace, makes decisions, and performs actions instead of just answering questions.
- Multimodal AI Creation: Integrated image, video, 3D model generation, and procedural visualizations into one spatial workspace.
- Inclusive Spatial Design: Added high-contrast visuals, readable interfaces, comfortable layouts, and voice-driven controls to make XR more accessible.
What we learned
We learned that accessibility should be built into XR experiences from the beginning, not added later. Voice-first interaction can reduce physical effort and make spatial computing easier for everyone.
We also discovered that AI becomes much more powerful when it can understand and interact with its environment. Instead of simply responding to prompts, spatial agents can perform real actions, manage workflows, and become creative partners.
What's next for WeaVR
- More Accessibility Options: Add gaze-dwell navigation, switch access, customizable high-contrast modes, and audio descriptions.
- Smarter Spatial Agents: Improve context awareness and autonomous multi-step workflow creation.
- Collaborative XR: Allow multiple users to create and work together in shared virtual spaces.
- More AI Tools: Expand support for 3D generation, interactive visualizations, and creative workflows.
Our vision is to make spatial computing accessible by default, powered by intelligent AI agents, and controlled through natural conversation.
WeaVR - Weave Anything.
Built With
- iwsdk
- runway