About the Project

Inspiration

Our inspiration came from our experiences working with people with disabilities. Through these interactions, we saw how difficult communication could be for individuals with speech challenges and how much being able to express yourself can affect everyday life.

We also learned that early speech and communication practice can make a meaningful difference. However, traditional exercises can often feel repetitive, tedious, or intimidating, especially for younger learners. We wanted to create something that made people genuinely excited to practice speaking.

That led us to a question: What if improving your communication skills felt less like an exercise and more like an adventure?

We created a Greek mythology-inspired game where players explore Olympus, interact with gods, complete quests, and practice communicating through natural conversations. While accessibility was our initial motivation, we wanted to make the experience useful for anyone looking to improve their English, speak more clearly, or articulate their thoughts more effectively.

What It Does

Our project is an interactive, story-driven adventure that turns speaking practice into gameplay.

Players journey through five chapters of Greek mythology, interacting with characters such as Aphrodite, Athena, Ares, Hades, and Zeus. Instead of relying entirely on predetermined dialogue choices, players can speak directly to characters through their microphones.

The game also features an AI-powered Speaking Companion that provides personalized communication feedback throughout the adventure.

Players can select goals such as:

  • Improving English grammar and vocabulary
  • Expressing ideas more clearly
  • Practicing speaking pace
  • Developing clearer communication

After speaking, players can review their transcriptions and receive suggestions tailored to their selected goals. The companion also tracks practice time and estimates speaking pace.

Our goal is to create an engaging, accessible environment where players can develop their communication skills at their own pace.

How We Built It

We built the frontend using React, TypeScript, and Phaser 3. React handles the user interface, dialogue boxes, microphone controls, and Speaking Companion, while Phaser powers the 2D movement, environments, puzzles, and character interactions.

Our backend uses Python and FastAPI to connect the game to its AI systems.

For speech recognition, we integrated OpenAI's Whisper-1 model to convert recorded speech into text. Because our game includes uncommon Greek mythology names, we developed a vocabulary-aware transcription approach that provides Whisper with contextual hints.

We evaluated this approach using 90 recordings from three speakers. On our 30-recording held-out test set, word error rate decreased from 1.2% to 0.6%, a 50% relative reduction. Mythology-name recognition improved from 93.9% to 100%. These results are preliminary because our test set was small, scripted, and used provisional reference transcripts.

We also used large language models to generate context-aware NPC conversations and personalized speaking feedback. Each NPC has its own personality and knowledge of the story, while our game logic controls quest progression independently of AI-generated responses.

Additionally, we experimented with PyTorch, Hugging Face Transformers, and LoRA fine-tuning to investigate whether speech recognition could adapt to individual speakers. Although we successfully trained an experimental adapter, our results did not yet establish reliable speaker-specific improvement, so we retained the better-performing hosted transcription system for gameplay.

Challenges We Faced

One of our biggest challenges was balancing natural AI conversations with structured gameplay. We wanted players to feel like they were genuinely speaking with Greek gods, but we also needed the storyline, puzzles, and progression to remain consistent. To solve this, we separated AI-generated dialogue from the game's authoritative quest logic.

Another challenge was improving speech recognition for unusual names. General-purpose transcription models occasionally struggled with mythology-specific vocabulary, so we built a benchmarking pipeline to measure transcription errors and test whether contextual prompting helped.

We also encountered technical challenges integrating microphone recording, transcription, AI responses, and speaking feedback into one seamless experience. Making these components work without interrupting gameplay required careful coordination between our React frontend and FastAPI backend.

Finally, designing communication feedback for people with different learning needs required us to think carefully about accessibility. We wanted to support players with speech disabilities without suggesting that everyone should speak at the same speed or with the same accent. This motivated us to provide customizable goals, editable transcripts, and optional feedback instead of rigid speaking scores.

What We Learned

This project taught us that building meaningful AI applications requires more than simply connecting to a model.

We learned how to evaluate machine-learning systems using real recordings, measure word error rates, experiment with model fine-tuning, and distinguish genuine improvements from results that might be caused by small datasets.

We also gained experience integrating multiple technologies into a complete application, from React and Phaser gameplay to Python APIs and AI-powered interactions.

Most importantly, we learned that accessibility and engaging design can reinforce each other. By embedding communication practice into an adventure, we hope to make learning feel more approachable, motivating, and enjoyable.

What's Next

We hope to expand our Speaking Companion with more sophisticated audio-based feedback, including pause analysis and targeted pronunciation practice. We also want to improve personalization, support more languages, and evaluate the game with a broader range of learners and speaking abilities.

Our long-term vision is to make communication practice something people look forward to, whether they're learning English, working on public speaking, or navigating speech-related challenges.

We want players to leave Olympus not only having completed an adventure, but also having gained more confidence in expressing themselves.

Built With

Share this project:

Updates

Submission history