-
-
Folder structure of the code environment
-
Installed unity module
-
Project uploaded on the Unity hub
-
Text drawn with the AI summary generated via HuggingFace
🚀 Inspiration
With the rapid rise of spatial computing and XR, natural handwriting inside immersive environments is becoming essential. Traditional drawing tools capture strokes — but they don’t understand intent.
I wanted to explore how MX Ink could evolve from a simple stylus into an intelligent spatial productivity tool. The vision was to combine natural 3D writing with real-time AI understanding to make interactions inside Meta Quest smarter and more meaningful.
✨ What it does
This project enables users to draw naturally in a Meta Quest 3D environment using MX Ink and instantly generate an AI-powered summary of their handwritten content.
With a single key press, the system analyzes the drawing using the Hugging Face Inference API and returns a meaningful interpretation.
🔑 Key Capabilities
- ✍️ Real-time handwriting capture in 3D space
- 🤖 AI-based drawing interpretation and summarization
- 🎮 Smooth spatial navigation (W/A/S/D controls)
- 🧹 One-click canvas clearing with live status feedback
- ⚡ Fast and responsive XR interaction loop
🛠 How we built it
The immersive environment was developed in Unity, with interaction logic written in C#. Development and API integration were handled in VS Code, and the Hugging Face Inference API powers the AI summarization.
🔄 System Workflow
- Capture drawing strokes from MX Ink input
- Render strokes dynamically on the Unity canvas
- Convert the drawing into an image payload
- Send the image to the Hugging Face model
- Display the generated summary in real time
⚠️ Challenges we ran into
Building a smooth spatial-AI pipeline came with several challenges:
- Ensuring fluid and natural stroke rendering in 3D
- Converting freehand drawings into AI-ready input
- Managing real-time API latency
- Synchronizing movement and drawing interactions
- Maintaining performance inside the XR environment
Each challenge pushed us to optimize both the interaction pipeline and the AI workflow.
🏆 Accomplishments that we're proud of
- ✅ Successfully integrated MX Ink + Meta Quest + AI
- ✅ Achieved real-time drawing summarization
- ✅ Built a responsive and intuitive XR experience
- ✅ Designed a scalable foundation for spatial AI tools
- ✅ Demonstrated practical productivity use cases
📚 What we learned
This project provided deep hands-on experience in:
- XR interaction design in Unity
- Real-time stylus input handling
- Hugging Face inference pipeline
- Performance optimization in immersive apps
- Designing AI-assisted spatial workflows
Most importantly, we learned how AI + spatial computing together unlock powerful new interaction paradigms.
🔮 What's next
We see strong potential to evolve this into a full spatial productivity platform. Planned improvements include:
- ✨ Improved handwriting recognition accuracy
- 🖐 Gesture-based controls using MX Ink
- 👥 Multi-user collaborative whiteboard
- ⚡ On-device AI for ultra-low latency
- 📒 Integration with note-taking and productivity tools
- 🧠 Context-aware AI understanding of diagrams
MX Ink + AI is just getting started.
Log in or sign up for Devpost to join the conversation.