Speakworld

Inspiration

I’ve never been great at learning languages through flashcards or repetitive exercises, but I’ve always loved getting lost in games. That made me wonder if language learning could feel more like entering a new place: travelling around, meeting people, and learning because you actually need to communicate. Speakworld came from that idea.

What it does

Speakworld is an open-world language-learning game where players explore India, Japan, and Mexico and practise speaking with the people they meet. Each world contains ten missions based on situations someone might actually face, including ordering food, asking for directions, visiting a pharmacy, and buying a train ticket. A local guide first teaches the player a few useful phrases. The player then travels to the relevant character and practises the conversation through the OpenAI Realtime API. Players can explore on foot or use local transport such as scooters, bicycles, trains, subways, and metros. The rail systems continuously travel between stations and stop long enough for the player to board.

How we built it

I started Speakworld with no experience in Three.js, Blender, 3D modelling, or game development. I used Codex with GPT‑5.6 as my engineering partner throughout the hackathon. I described the experience I wanted, tested every version, and explained what felt confusing, empty, crowded, or simply not fun. Codex implemented those decisions in code. Through Computer Use, it also operated Blender and Chrome, generated and exported the environments, inspected visual problems, and tested the game as a player would. Together, we iterated on movement, collisions, missions, maps, characters, transport, sound, dialogue, and automated tests. Codex also integrated OpenAI’s Realtime API, kept the API key server-side, and added speech detection, subtitles, and different character voices. Speakworld uses Three.js, JavaScript, Vite, Blender, WebRTC, Web Audio, and OpenAI APIs. I directed the product and decided how the game should look, teach, and feel; Codex and GPT‑5.6 helped me turn those decisions into a working game.

Challenges we ran into

Since I had no experience with web development, Blender, Three.js, or game development, one of the hardest parts was explaining what I imagined and then testing whether it actually felt right. I worked by playing each version, spotting problems, and asking Codex to improve them. The 3D world took a lot of iteration. Roads could look correct in Blender but feel uneven in the game, vehicles could get stuck, and the camera or collisions sometimes made exploration frustrating. Natural voice conversation was another challenge. Early versions interrupted hesitant speakers, cut off longer NPC replies, or started feedback before the character had finished talking. We had to improve turn detection, conversation limits, audio timing, and microphone behaviour. It was also difficult to make three large worlds, transport systems, missions, sound, and live conversations feel like one complete game within the hackathon deadline.

Accomplishments that we're proud of

I’m proud that Speakworld became a playable open-world game, even though I began the hackathon with no experience using the tools needed to build it. Players can explore three different countries, complete 30 missions, travel by scooter, bicycle, train, subway, or metro, and practise live conversations with local characters. The transport systems move continuously through each world and stop at stations long enough for the player to board. What I’m most proud of is the learning flow. The guide teaches the player a few useful phrases, the player travels through the world to find the right character, and then uses those phrases in a real conversation. It makes learning feel connected to a place and a purpose rather than separated into exercises. I’m also proud that the different parts feel like one experience rather than several disconnected demos. The environments, missions, navigation, transport, character dialogue, sound, cached guide lessons, and Realtime conversations all work together inside the same browser game.

What we learned

I learned that I could build something far outside my existing skills if I could explain the idea clearly, test it honestly, and keep improving it. I didn’t learn Blender or Three.js in the traditional sense, but I learned how to direct Codex across unfamiliar tools and turn an idea into a working product. I also learned that language practice becomes more memorable when the words have a purpose. Asking for a ticket while standing at a station feels completely different from selecting the same sentence in a quiz. Voice learning also depends on small details. Learners need time to think, make mistakes, finish their sentences, and understand why a character responded in a certain way. The technology should make them feel comfortable speaking, not pressured.

What's next for Speakworld

Next, I want to make each country feel even more distinct through its architecture, transport, sounds, characters, social customs, and conversation styles. I also want to add more countries, longer storylines, saved progress, changing weather, and conversations that remember what the learner struggles with. Custom practice could become a bigger part of the game, allowing someone to prepare for a job interview, hotel check-in, restaurant visit, or upcoming trip. The long-term goal is for players to arrive knowing very little, then gradually become confident enough to explore and speak without following a prepared script.