Inspiration
We were inspired in particular by https://oasis-ai.org/, which replicates games purely using generative AI to simulate a frame-by-frame experience. However, we wanted to amplify the experience beyond simply an "AI-built" game.
What it does
EmotionAI takes a standard 2D platformer and breaks tradition in two ways. First, it is a game primarily built by generative AI. As the player's emotions and stress levels are measured and change states, a generative AI model revamps the setting of the world, including "game-specific" components like platforms and water (when you're angry, the lava will send you flying!). These changes are caused by the player's emotions, demonstrated through their facial expressions. This adds a whole new dimension to gaming!
How we built it
We started with a basic 2D game engine to generate chunks, along with an integration with a computer's webcam. The data from the webcam was combined with a pretrained CV model to determine the probability distribution among several emotions, from which we derived a custom "stress" metric. The emotion and stress metric were then used to alter the game engine assets via a generative image model to generate objects that represent the emotion felt (and also create textures for level objects like platforms!).
Challenges we ran into
Our original idea was more focused on a full generative AI aesthetic, where images were generated frame-by-frame—however, after implementing this design, our model took 1-2 seconds per frame, which would have been unplayable. Another major challenge was integrating the continuous data from the webcam with the large AI image-generation model. Because these modules were so computationally heavy, it was difficult to strike a balance between high speed and quality graphics. We ran into a lot of trouble integrating these components together, as they all relied on each other.
Accomplishments that we're proud of
We were really proud of not only the concept, which has never been accomplished before, but also the amount of work we were able to produce in a short time with hardware limitations. Without advanced GPUs and strong computers, being able to create a game that reliably generated visual graphics was really motivating.
What we learned
We really learned how to build using Replicate and general image generation techniques, while also connecting AI, hardware-ish, and software components into a cohesive, functional app. Furthermore, our hardware constraints pushed us to discover more lightweight image generation techniques, as well as sharing resources like the camera between the calibration and core game loop.
What's next for EmotionAI
Integration with enemies—making it even more of a platformer! Given more time, we'd like to implement a frame-by-frame image generation technique instead of our current chunking approach.
Log in or sign up for Devpost to join the conversation.