Inspiration
When I first learned a language, covering my room with sticky notes helped me remember vocabulary. I often wished I could automatically generate these notes, hear pronunciations, test my own speaking, and see the words used in sentences. When I learned about Meta’s Camera Access with context-aware AI, I realised I could finally build this, but far better than physical notes, and in a way uniquely suited to Quest’s mixed reality capabilities.
What it does
Home Fluent is a mixed-reality language tutor that transforms your real environment into an interactive learning space. First, the app detects objects around you using Meta’s context-aware AI and anchors vocabulary labels directly onto them. Tapping a label lets you hear a native-quality pronunciation (via TTS), practice saying the word (graded via speech-to-text), or generate contextual example sentences using an LLM, all in real time. This creates repeatable, personalised learning moments tied to the player’s actual environment, something only possible on Quest.
Second, Home Fluent includes a fast, replayable Word Search mode designed for “time to fun.” It selects a word you learned that session and challenges you to physically move around your room to find the matching object within 60 seconds. It’s energetic, memorable, and encourages revisiting sessions.
While there are other language-learning experiences on the Quest store, including one in MR, Home Fluent is different in that it automatically uses your actual space; and its deep integration of AI in multiple locations means that the size of your learning vocabulary is not limited by pre-programmed word lists; while the ability to generate unlimited sentences for you to try your learning in adds even broader scope than existing offerings.
How we built it
We began with Meta’s Camera Access example project, then optimised how we used it for near real-time comfort and performance on Quest hardware. We integrated three AI systems - dynamic TTS, speech-to-text pronunciation scoring, and LLM sentence generation -while maintaining a smooth 60+ FPS on Quest 3/3S. Once stable, we layered in game logic, UI flows, and polished MR visuals to create a clean, intuitive UX.
Challenges we ran into
The biggest challenge was balancing real-time object recognition with performance. Constant polling dropped framerate, so we redesigned recognition polling and caching to maintain responsiveness without sacrificing accuracy or user comfort. Achieving fluid MR interactions while juggling multiple live AI systems required careful optimisation and iteration.
Accomplishments we’re proud of
Home Fluent meaningfully showcases what Quest can do that other platforms can’t: a personalised, space-aware language tutor that blends productivity, play, and mixed reality. And yes - my Spanish is genuinely improving from using it!
What we learned
We gained a deep understanding of the context-aware AI’s capabilities, its performance trade-offs, and how to integrate several AI modalities while preserving UX quality.
What’s next
We plan to expand long-term learning features, add spaced repetition, support more languages, and explore conversational MR interactions that adapt even more deeply to the player’s environment.
Log in or sign up for Devpost to join the conversation.