Immersive 3D Whiteboard for VR

Inspiration

Traditional collaboration tools still feel flat. Brainstorming on 2D screens often limits creativity, especially when working remotely. Our team wanted a way to think spatially, collaborate naturally, and build ideas as fast as we speak them.

What it does

Our system lets multiple users enter a shared VR space and collaborate on a large 3D whiteboard. You simply speak a prompt (“Map out our sprint plan”, “Break this problem into tasks”), and our AI pipeline interprets your voice, converts it into nodes and connections on the whiteboard, generating a well-structured flowchart. Gestures and spatial controls let users move, resize, or link elements.

How we built it

WebSpatial (Vision Pro): Used to create a fully spatial, gesture-based 3D whiteboard space

ChatGPT Realtime API: Powers continuous, natural conversation with the user and interprets spoken instructions

JSON Workflow Builder: Converts AI-parsed instructions into a standardized graph-like JSON format

tldraw AI Generation: We passed JSON directly into tldraw’s AI to generate accurate shapes, arrows, and flowchart components

Flowchart Assembly: The elements from tldraw render dynamically in the whiteboard, creating a clean and editable workflow diagram

Challenges we ran into

Like any typical hackathon, we encountered numerous issues. One of the biggest challenges we faced was getting our entire pipeline — WebSpatial, the ChatGPT Realtime API, and tldraw’s AI generation — to work smoothly inside the VisionOS environment. Setting up WebSpatial required carefully restructuring our project files and assets so the spatial scene loaded correctly. Additionally, implementing the code for VisionOS became significantly more challenging because we lacked access to a physical Vision Pro headset. Relying on the simulator meant we couldn’t use the microphone, which broke our real-time voice interaction flow and forced us to mock or bypass essential features. These limitations led to debugging blind spots and made assembling the full end-to-end experience far more complex, especially when trying to validate 3D placement, performance, and audio-driven workflows without real hardware.

Accomplishments that we're proud of

We're proud that we were able to build an end-to-end system where spoken ideas transform into structured flowcharts inside a spatial 3D environment. Even with limited hardware access, we successfully integrated WebSpatial on VisionOS with the ChatGPT Realtime API and built a JSON pipeline that tldraw’s AI generator could convert into real flowchart elements. Getting all these components — voice interaction, AI parsing, JSON formatting, and 3D rendering — to work cohesively was a major accomplishment. We also proved that conversational creation in mixed reality feels both natural and intuitive, validating our core idea even without a physical Vision Pro device.

What we learned

We learned just how powerful voice-driven interfaces become when paired with spatial computing and AI. Building this project taught us the importance of structured intermediate formats which in our case was JSON, for bridging natural language and visual diagram generation. We also learned a lot about the VisionOS ecosystem, including the limitations of working in a simulator and how critical real hardware is for accurate spatial design and audio input testing. On the AI side, we better understood how to shape prompts, detect intent from streaming speech, and convert conversational phrases into a predictable graph structure. And above all, we learned that multimodal interactions (voice + spatial visuals) create a surprisingly intuitive workflow creation experience.

What's next for Immersive 3D Whiteboard for VR

Next, we plan to add an AI-powered suggestion box that analyzes the user’s ideas and recommends additional nodes, improvements, or follow-up steps. Users will be able to approve or deny each suggestion, and once approved, the system will automatically draw the new element and connect it to the existing diagram. This feature will be powered by the ChatGPT-5 model, giving the whiteboard the ability to intelligently expand workflows, identify missing steps, and enhance brainstorming sessions. We also want to introduce an AI tutor mode inside the whiteboard. In this mode, the system can explain concepts, break down tasks, guide a user through problem-solving, or help them understand the structure of their flowchart. Combined with spatial visualization, this turns the whiteboard into an interactive learning assistant.

Built With

• css
• html
• javascript
• python
• typescript
• visionos
• webspacial
• whisper