-
-
Login screen
-
Connection screen
-
Talking to another NPC!!
-
Talking to an NPC!
Inspiration
I drew inspiration from "Nothing Forever", An animated 3d series who paritied the show "Seinfeld" but all the graphics were computer generated and the dialogue was completely generated by LLMs and Audio Generation models. I also drew inspiration from games such as Valorant who also use a similar network protocol model as I did to ensure that nobody can abuse client modifications.
What it does
Verbalize is a 3d social networking platform that allows you to experience the world as though the world was ecapsulated in an 1830's Steampunk era village right in your web browser! Verbalize allows you to naturally speak to the inhabitants within these villages while also being able to enjoy the experience with your friends as well. These villagers can also perform specific tasks for you using their given skillset and tools!
How we built it
Initially, I designed a large specification of what I wanted for the network protocol (Protobuf + compression using deflate), frontend libraries, and a basic demo renderer using raw WebGL, alongside API docs for the Gemini API and ElevenLabs to initially test multiplayer. Then, through recursive prompting, iteration on the frontend, and manual review & engineering, I achieved a result that I visually liked and kept. The Kotlin/Ktor backend was also written under strict supervision to follow exactly how I wanted it to be structured (Heavy DDD, but not touching Spring Boot because nobody likes Spring Boot + serverside authority over every action the client does, and my choice to use MariaDB to store user information), and was edited in post to fix some issues that Claude did not pick up on.
Challenges we ran into
The main challenge I ran into was really making sure systems stayed in sync. For example the serverside physics simulator (for our serverside authorative movement), originally the tiniest amount of latency would cause "rubberbanding" (server force resyncing the client's position) due to small mismatches in what the server calculated and what the client predicted. Another issue regarding this was also ensuring that a race condition didn't occur when 2 people were talking to a model at the same time causing the memory of the model to be out of order/corrupt.
Accomplishments that we're proud of
The two parts that I am really most proud of is getting the serverside movement and clientside prediction to be smooth enough to be innoticable and also the fact that I was able to get Gemini to be able to respond fast and implementing my own solution for ElevenLabs to stream the generated audio over the network instead of waiting for it to fully download so that generated verbal responses feel near instantanious & natural. Furthermore integrating Google's Mediapipe library into the project for facial feature detection pushed me out my comfort zone in terms of development, but in the end I really liked the outcome of it.
What we learned
I learned that software development isn't just the ability to write code, it is also the ability to quickly think outside the box & create solutions that are novel and even messy.
What's next for Verbalize
I would really like to continue improving Verbalize by making the backend even more efficient than it currently is and further optimizing the game renderer. Afterwards I would like to improve on features such as letting the villagers be able to converse with themselves (like "Nothing Forever"), but ensure that they do not waste too many tokens on it. I also thought of allowing users to save and load worlds locally without the need of a remote server and also allowing for players to load homes that were built for them into other rooms.In general, I would make the world more interactive.
Log in or sign up for Devpost to join the conversation.