AI-powered Immersive Japanese Language Learning Game in VR

interactive dictonary
AI Chat, Nightcity scene
AI Chat, Sakura street scene
AI Chat, Convenient store scene

Inspiration

As a passionate language learner with a particular interest in Japanese, I've experienced firsthand the challenges of traditional language learning methods. While textbooks and classroom learning provide essential foundations, they often lack engagement and practical application. Although immersive environments like studying abroad offer ideal learning conditions, they require significant time and financial investment. Even when in Japan, language anxiety and lack of constructive feedback can hinder actual practice. My journey with various learning tools has been enlightening. Gamified platforms like Duolingo keep learners engaged but sometimes sacrifice learning depth. AI tools like ChatGPT offer powerful personalized assistance but lack immersion. VR applications like Mondly VR provide immersive experiences but often miss structured learning progression. These experiences inspired me to create a solution that combines the best of all three: the engagement of games, the adaptability of AI, and the immersion of VR.

What it does

Our platform creates an AI-powered Japanese learning experience in virtual Tokyo. The system has three main connected features:

The Scene Interaction system acts like an interactive dictionary in VR. When users point at objects - from street signs to menu items - they get instant translations, pronunciations (in both romaji and hiragana), and example sentences. This helps learn vocabulary naturally in the right context.

The heart of our system is the AI Chat feature. Users can practice Japanese with different AI virtual NPCs in various scenes. The AI adapts to each user's level, providing relevant feedback and suggestions while keeping conversations flowing naturally. Users can practice real daily Japanese conversations, from asking for directions to ordering coffee, helping them build language skills and speaking confidence.

We've also added game elements to make learning more engaging. We've incorporated game elements including a life system with hunger and fatigue mechanics, and a virtual money system where users can work as a café barista to earn currency for translation assistance. The system encourages learning through actual use rather than just memorization.

How we built it

The development process began with extensive user research, interviewing Japanese language learners about their learning challenges and needs. After creating initial wireframes and prototypes, we refined our design through multiple iterations of user feedback.

For technical implementation, we used Unity 2022.3.55 on a PC with Intel Core i7 processor and NVIDIA RTX 3080 GPU. The virtual environments were constructed using assets from Unity Asset Store and Sketchfab, while NPC avatars were created through Ready Player Me. We chose Meta Oculus Quest 3 as the primary VR device, while maintaining a desktop version for accessibility.

The technical implementation phase focused on Unity development for VR environments, followed by AI integration. We built the dialogue system by combining multiple APIs: Claude 3.5 Haiku Model for generating dynamic NPC responses, OpenAI Whisper for speech recognition, and Azure Speech Studio for natural Japanese voice output. The game mechanics were carefully designed to enhance learning engagement while maintaining educational value, drawing inspiration from life simulation games for progression systems and reward mechanics.

Challenges we ran into

One of our biggest challenges was getting the AI conversations to feel natural while keeping the VR experience smooth and responsive. Making multiple AI systems (Claude, Whisper, and Azure) work together seamlessly in VR took a lot of trial and error. We spent considerable time fine-tuning our AI prompts and conversation system. It was challenging to structure prompts that would generate responses that are not only natural and contextually appropriate, but also educational - providing language learning opportunities at just the right level for each user. We had to carefully design how we parse user inputs and format AI responses to maintain conversation flow while extracting useful learning patterns and providing helpful feedback. We also spent considerable time creating authentic Japanese scenarios that are both fun and educational - it's a delicate balance between making something entertaining and ensuring it actually helps people learn. Additionally, we had to optimize everything carefully to maintain high-quality VR environments without compromising performance or making the controls too complicated.

Accomplishments that we're proud of

We successfully created a system that seamlessly integrates AI technologies, VR environments, and gamification elements. Our dynamic dictionary system leverages spatial memory principles for enhanced vocabulary retention. The AI chat system provides personalized, context-aware conversations with natural voice interaction. We're particularly proud of implementing authentic Japanese scenarios with culturally appropriate interactions and creating an engaging progression system through gamification features like the virtual job system.

What we learned

This project taught us a lot about creating educational experiences that are both fun and effective. We learned how to optimize VR applications and handle complex AI systems, but more importantly, we discovered how crucial cultural authenticity is in language learning. We also gained insights into how technology can help overcome both technical challenges (like providing instant feedback) and psychological barriers (like anxiety about speaking a new language).

What's next for AI-powered Immersive Japanese Language Learning Game in VR

We're planning to conduct extensive user testing to see how people actually use the system and what they find most helpful. We want to add more scenarios and make the conversations even more natural. We're also excited about adding multiplayer features so learners can practice together, and possibly adapting the system for other languages. There's potential to use this approach in professional training or other educational settings too. Most importantly, we'll keep refining everything based on user feedback to make the experience even better.