iClicker Plays Pokemon

Chaotic collaborative gameplay of Game Boy games using multiple iClicker devices.

Comment

Note: For the Documentation Prize, please check out our complete wiki guide!

Inspiration

Our inspiration for this hardware hack was the popular online phenomenon Twitch Plays Pokemon, a 2014 social experiment which allowed millions of people of online to crowd-source the effort to beat well-known Pokemon games, one button input at a time. The idea came when we considered how this could work if we tried to do the same in the physical context of a lecture hall, just using the iClicker devices that are used by many classes on our campus.

What it does

Our goal for this project was to create a fully modular and open-sourced API for using iClicker devices outside of their proprietary polling software, and to then develop a working application that used the iClicker data to play emulations of GameBoy games in real time. The iClicker hardware supports up to 1500 iClicker devices, and can receive up to 200 inputs per second. We created a working demo that allows gameplay of six games, with support for as many ROMs as desired.

How we built it

The open-sourced iClicker Plays Pokemon project was built on top of Python, JavaScript, WebSockets, and Django. Our demo was created in addition using a modified JavaScript GameBoy emulator and the Materialize CSS library.

Challenges we ran into

We had planned to be capture button-presses off the D-pad buttons on the iClicker remotes, just like we capture the A-E buttons. However, we discovered that in the special hardware mode that allows the D-pad to function, the D-pad buttons only control configuration on the local devices about what information to send, and we cannot capture when they are directly pressed. We were able to use these buttons to support inputs of any alphanumeric character (such as "F", for which a physical key doesn't exist) in order to support more than 5 possible inputs to the game, but it was disappointing that the physical D-pad couldn't be used intuitively like on a gamepad.

The JavaScript Game Boy emulator that we used to render the games was annoyingly difficult to adapt into our project, due to both the poor coding style of the developer -- such as using XHTML self-closing tags in non-compliant ways, and polluting the global namespace -- and the tie-in between the emulator itself and the developer's intended GUI around it, which we had to manually remove.

Accomplishments that we're proud of

Using a dormant GPL'd Python project designed for an older version of Python, which used pyusb to support some basic features of the iClicker remotes by scraping the USB communication of the base station, we were able to decode nearly all of the functionality used by the iClicker desktop application to communicate with the remotes. The alphanumeric mode, which allows for the D-pad buttons to send any string of alphanumeric characters to the base, not only allows for more than 5 inputs from an individual iClicker, but also cuts down on the latency between button presses from around 750ms to 250ms. This results in a better experience with no input lag to the emulator. This was all done with no protocol documentation, through reverse-engineering the USB communication with the base station.

Additionally, we were able to incorporate this USB sniffing into a Django project, that serves a full web application with websockets communication, and incorporate sending websocket commands into the callbacks for USB reads which allow the client web browser to be informed of the hardware button presses from the device.

What we learned

With less experience with combining multiple different libraries and resources together, I (Chris) learned a lot about the difficulty (and simplicity) of combining APIs together in JavaScript project. With iClickers the information being received via USB connection isn't algorithmically or high-complexity, but I learned how the JSON objects could be used quickly to handle streams of data with varying size and type. I also learned how to use wikis on GitHub for creating documentation and how to set up web servers using redis and Django. Lastly I learned that integrating USB and other external devices (like the iClicker Base Station) can be difficult between target platforms and will generally require different libraries, since libusb didn't work on Mac, but it could be adapted using cython-hidapi instead.

What's next for iClicker Plays Pokemon

Since our API has been modularized so just the Python/WebSocket module can be used separately, our tool can hypothetically be used for any JavaScript / web-based solution for polling students or creating games for users, even those that aren't necessarily real-time like GameBoy games. We have created complete documentation on the GitHub repository for our code that explains information about hardware, installation, troubleshooting, and using the API.

Built With

Share this project:

Updates