Inspiration
I study Computer Engineering in Argentina. When preparing for an exam, I end up with notes, formula screenshots, diagrams, and several open tabs. When I go back to study, part of the work is just finding where I left everything.
I wanted to distribute that material around my physical desk and find it again in the exact same context. I also wanted to use it to quiz myself and practice actively, rather than just rereading. I took inspiration from the method of loci (memory palace), which associates ideas with familiar places. In RoomCanvas XR, your room remains visible: the desk is where I study, and around it, I can place a formula, a diagram, or a question.
What it does
RoomCanvas XR is a mixed reality spatial study app for Meta Quest that runs directly from the browser. You can start with pre-loaded examples or send your own images and flashcards from your phone or computer using the Companion app.
Inside the experience you can:
- Place and resize panels containing notes, formulas, images, and diagrams.
- Create or edit Q&A flashcards.
- Review using a shelf based on the Leitner system: try to answer, reveal the solution, and log your review to schedule the next practice session.
- Explore interactive learning modules. As a proof of concept, the app includes an interactive derivative graph where you can manipulate the secant line and watch it approach the tangent, demonstrating how abstract mathematical concepts can become visual and tangible right in your room.
Spaces save your material and progress so you can resume exactly where you left off in another session.
How I built it
I used Meta IWSDK and WebXR, alongside TypeScript and Three.js for the immersive experience. The Companion app is built with React and connects to a Python backend using FastAPI. I chose the web approach so the prototype could be launched via a simple link in the Quest Browser, with no app installation required.
Formulas are rendered using MathJax as SVGs inside XR, and KaTeX in the Companion app. Space data is saved locally in the browser, while images sent from Companion are stored on the server and retrieved via authenticated requests.
Meta VR Features & Interactions: I heavily utilized hand tracking, passthrough, spatial computing (surface/anchor detection via WebXR available capabilities), and spatial audio. Passthrough is crucial as it allows studying with virtual notes while staying aware of your physical room. For hand interactions, the thumb-index pinch allows users to select and move cards both up close and from a distance. On panels, grabbing the frame moves the entire set, while specific graph controls allow direct interaction.
Challenges I ran into
Saving a panel's content and recovering its spatial position in the room are two entirely different problems. The headset might go to sleep, lose tracking, or return to a session with a different spatial reference. I had to work extensively on verifying poses before saving to prevent pending operations from corrupting another space.
Math formulas also brought UI challenges: a short expression and a multi-line equation don't fit the same way. I had to adjust rendering, layout, and pagination dynamically. Another major hurdle was separating the user's identity from the temporary pairing PIN so that renewing a PIN wouldn't break the connection to the images the user had already uploaded.
Accomplishments that I'm proud of
Taking a genuine need from my own study routine and turning it into a real, testable prototype. I love that it allows users to work with their own material, write their own review questions, and interact with abstract concepts (like the derivative prototype) right next to their notes. I'm also very proud of the technical milestones: seamlessly saving flashcard edits and updating Leitner progress without the panels losing their physical position in the room.
What I learned
I learned that resuming a session requires just as much UX attention as starting one. An out-of-sight button or a panel that randomly shifts can completely break a student's focus. I also learned to distinguish between what automated tests prove and what must be verified with the headset on: an interaction passing a code test doesn't mean it actually feels comfortable, natural, or clear when using your hands in MR.
What's next for RoomCanvas XR
I want to test the prototype with more students to observe what material they import, how they organize it spatially, and where they face friction. I also want to add a feature allowing teachers to share "study packages" while preserving each student's individual progress and spatial layout.
The current interactive module is just a starting point. In the future, I want students to be able to request activities via a prompt in Companion (e.g., 'I want to understand how the slope of this function changes'), allowing an AI to instantly generate interactive graphs, diagrams, or simulations and send them directly to Quest to be manipulated in their spatial environment.
Initially, I considered using AI to automatically generate notes or summaries, but I realized that if the tool does all the heavy lifting, the student misses out on a vital stage of the learning process. My vision is that AI in RoomCanvas XR shouldn't replace the effort of studying; instead, it should represent a commitment to actual learning—building scenarios for deep comprehension rather than shortcuts to avoid thinking.
Finally, post-hackathon, I plan to work on local image storage within the headset and encrypted, asynchronous synchronization. RoomCanvas XR is an actively evolving project, and I highly value professional input; any recommendations, technical feedback, or collaboration proposals to help scale this vision are always welcome.
