Gamified Language Learner

Inspiration

Inspired by the concept of historical "Dragomans" – cultural interpreters who guided travellers through unfamiliar lands – our project aims to transform language acquisition from textbook rote memorization into an immersive, interactive experience. We sought to create a virtual training system that blends practical language skills with crucial social and cultural norms, making learning engaging and immediately applicable.

What it does

Gamified Language Learner" (GLL) is a 3D interactive game where players embody a tourist embarking on their first international trip. The core of GLL involves navigating various real-world scenarios – from an airport arrival to a taxi ride, hotel check-in, and ordering at a cafe – by engaging in natural conversations with AI-powered NPCs. Players choose dialogue options, interpret NPC responses (presented in both the original language and with translated subtitles), and receive immediate feedback and "XP" based on their accuracy and cultural understanding. The game fosters not just linguistic proficiency, but also confidence in handling diverse social interactions.

How we built it

We built GLL as a React application, leveraging @react-three/fiber and @react-three/drei to render a dynamic 3D environment. For conversational AI, we integrated OpenAI's services for robust NPC dialogue generation, speech-to-text transcription, and text-to-speech capabilities, allowing for realistic voice interactions. The backend, powered by Express.js and socket.io, handles server-side logic and lays the groundwork for potential multiplayer interactions. Frontend styling was efficiently managed using Tailwind CSS. This combination allowed us to create a rich, interactive, and responsive language learning platform.

Challenges we ran into

One of the primary challenges was seamlessly integrating the various AI services (speech-to-text, text-to-speech, and dialogue generation) to create a fluid and natural conversational flow within the 3D environment. Ensuring real-time performance and managing API latencies for a smooth user experience required significant optimization. Additionally, designing the dialogue trees and scenario logic to provide meaningful learning feedback while maintaining engaging gameplay was a complex task, balancing educational goals with game mechanics.

Accomplishments that we're proud of

We are particularly proud of creating an engaging and functional proof-of-concept for immersive language learning within a hackathon timeframe. The integration of 3D graphics with advanced conversational AI to simulate realistic interactions is a significant achievement. We're also proud of the dynamic feedback system that goes beyond simple right/wrong answers, rewarding players for cultural nuance and confident communication. The foundation for multiplayer interactions also excites us as a future possibility.

What we learned

This project provided invaluable experience in developing full-stack applications that combine 3D rendering, real-time AI integration, and complex state management. We deepened our understanding of React's ecosystem, particularly in handling 3D scenes. We also gained practical insights into optimizing API calls for latency-sensitive applications and designing game mechanics that effectively facilitate educational outcomes. The iterative process of scenario design and dialogue scripting taught us the intricacies of crafting engaging language learning content.

What's next for Gamified Language Learner

Next steps for GLL include expanding the variety of locations and scenarios, introducing more complex cultural interactions, and developing a more sophisticated adaptive learning engine. We plan to refine the dialogue system to allow for even more free-form player responses and incorporate advanced pronunciation feedback. Further development of the multiplayer aspect to enable collaborative learning and competitive challenges is also a key goal. Ultimately, we envision GLL evolving into a comprehensive platform for immersive, culturally-rich language