For this hack, we wanted to create something fun and interactive for everyone. We've seen various integrations of community-driven games before, but we wanted to extend the idea further.
What it does
XPlaysX allows people to use a range of different platforms to send controller inputs of a range of different video games. The video feed is then streamed direcetly to twitch, for everybody to see the result of their shared gameplay experience. Whether you're using Twitch, Twillio, Twitter, Discord, Google Home, even Fiverr (you can't have a video game in 2019 without microtransactions), you can control the game. We even used the Capital One api to randomly generate movements!
With our web interface, admins can select and upload roms to change the game being played. Pokemon too overused? That's fine! How about some Zelda?
How we built it
In order to make a platform we could really easily add new integrations to, we abstracted our virtual emulator through RESTful requests, facilitated my a microservice architecture. It's really quite simple:
Each integration (Twitter/Discord/etc) runs on its own microservice and listens for inputs from users of that platform.
Inputs are then sent to a queue service which sends individual controls, through a RESTful endpoint, to our controller service. (This is to control the speed at which we're sending requests and avoid overloading our system.)
Our emulator service then receives these requests and sends them as inputs into a video game running on an emulator.
The emulator service is constantly streaming the emulator window as video output through FFMPEG to Twitch, over RMTP protocol.
Challenges we ran into
The entire emulation and streaming process was very challenging for a few reasons.
It took a while for us to learn how to send commands to our emulator.
It took even longer for us to figure out FFMPEG and getting our stream to run.
When we finally got the stream running, the delay from our emulator to twitch was about 15 seconds. We put in a lot of effort, manipulating bitrate, presets, and buffersize, to bring that down to 5 seconds.
Managing the state of games was also quite challenging, we needed to simulate and automate a user's experience fully because the emulator has no command line API.
Our Twitter integration proved quite challenging because the response was streamed as a constant stream of bytes, and we had only prepared for basic RESTful operations. We also needed to overcome a few authentication errors with twitter.
Accomplishments that we're proud of
All aspects of our sytstem: the bots, the emulator, the web frontend, everything is running on Google Cloud Platform. This is all of our first times using Google Cloud, and we're pleased with out it turned out. Some of the game emulation and video encoding can be process heavy, but the power of GCP made it possible, as well as their extremely fast network.
We were a bit shocked when, after hours of messing around with FFMPEG options, expecting nothing to happen, our Twitch stream suddenly burst to life. We suddenly felt a lot closer to finishing our hack.
We know "hacks" are supposed to be thrown together, but our architecture actually ended up being quite stable. Developing microservices allowed us to "throw together" bits of the system by bringing up new services and have them work together quite quickly. This also meant we divide the work up effectively and each work on independent parts of the project.
We didn't initially plan to use Python, but it ended up being a vital tool for our success, and we're quite proud of how we're making use of Python's vesitiliy in our project. We use it to control the Win32 Native API to send commands straight to our emulator.
What we learned
A couple of us had tried to use FFMPEG before for previous projects and failed horribly. It's a very challenging software suite to use, with a large learning curve. This hack forced us to overcome our inhibitions, get deep into the documentation, and understand how we needed to use it in our project.
How to inject keypresses with the Win32 Native API straight into processes. This is a skill that will be useful for many future automation projects.
Don't share git branches or you'll spend the next 2 hours rectifying it and the next 4 hours regretting it.