Inspiration
The idea for Niyo came from a simple frustration: managing tasks across different screens felt messy and distracting. I wanted an assistant that lived in my actual surroundings—something I could talk to, gesture at, and place anywhere in my room. The Meta Quest's hand tracking made that vision feel possible. The thought of pinning notes on walls, setting floating timers, and tagging physical objects without touching a device is what sparked the entire project.
What it does
Niyo is a fully hand-tracked Mixed Reality assistant that blends AI and spatial computing. You can:
- Create sticky notes that stay in your room
- Set floating timers using natural language
- Tag real-world items and find them through visual path lines
- Navigate onboarding slides using simple gestures
- View all notes, tags, and timers in a real-time dashboard
All interactions happen through voice or pinch-based gestures, making the experience almost completely hands-free.
How we built it
We built Niyo using Unity 6.2 combined with the Meta All-In-One SDK 81.0 for hand tracking, spatial anchors, and Quest-specific integration. The AI system is powered by GPT-4o, which interprets voice input and returns structured actions such as creating notes, timers, tags, or locating items. At the core of the project is a NiyoMasterController, which coordinates major systems:
OpenAI Manager for voice processing, intent extraction, and responses Database Manager for JSON-based data persistence and versioning PathFinder for visual guidance to tagged objects Timer and Note Controllers for spawning and updating MR objects Home Dashboard Manager that reflects real-time changes Walkthrough Controller for onboarding and teaching gestures
The workflow was built around making the assistant feel natural: speak a command → AI interprets it → the corresponding MR object is created, updated, or found. Every action is auto-saved, and duplicate prevention is handled with timestamps and cooldown logic. A version control system ensures that if the data format changes, older saves are safely reset to avoid corruption. Challenges we ran into
Early builds deleted all notes at once due to global pinch detection Spatial anchors sometimes caused MR objects to shift or reset Pathfinding lines occasionally rendered pink due to shader issues AI misunderstandings created unintended tags Old data appeared after installing updated builds Duplicate tag creation during rapid AI commands
Accomplishments that we're proud of
Built a 100% hand-tracked MR assistant with no controllers Achieved smooth, natural AI conversations with ~95% intent accuracy Reduced delete interaction time from 2.5s to 1s Eliminated false positives for AI listening Implemented reliable visual guidance and automatic data migration Delivered a minimal, clean dashboard that updates instantly
What we learned
We learned that gesture-driven MR requires forgiving UX design, with clear visual feedback to show the system's state. Spatial persistence is trickier than expected, and local world-coordinate storage turned out more reliable than anchors. Natural language makes the system feel more human, and thoughtful timing, cooldowns, and visual indicators significantly improve usability. What's next for Niyo – Your Personal Assistant
Multi-user shared MR spaces
Cloud sync across devices Smarter reminders and location-based triggers Checklists and task progress tracking Voice customization and accents Gesture training mode for new users
Log in or sign up for Devpost to join the conversation.