I was inspired when I saw Holly playing Legend of Zelda form her phone on a projector, as well as by the famous Twitch Plays Pokemon.

What it does

My hack comes in two parts: a web app and the backend server.

The web app functions as a controller that can control a game.

Note: The app is not functional on desktop as it requires the browser to support the TouchEvent APIs. I made the decision to remove support for desktop in order to improve the experience for mobile devices, since they are the target of the controller anyway.

How I built it

I built both the frontend and backend in Rust. I made use of WebAssembly as a compilation target and using the Yew framework (which has a similar component model to React).

By using WebAssembly I was able to make use of the excellent safety and speed of Rust and I was able to avoid the nightmare of writing in JavaScript. This had the benefit of being both easier to write and running faster than the equivalent JavaScript too.

Seriously check out Rust, it's great.

The backend makes use of RetroArch to emulate Pokemon Emerald, however all Game Boy Advance games are supported. Using the low level capabilities of Rust I was able to write an input driver in order to control the game.

I make use of a WebSocket to communicate to communicate between the web app and the backend server. The response time is very quick and makes the controller feel good to use.

Challenges I ran into

I encountered issues with duplicated inputs and poor touch support, which I worked to solve; as a result I gained a better understanding of the new web TouchEvent APIs.

I also ran into a challenge with the tungstenite library for managing the WebSockets on the backend; I unfortunately encountered a previously unreported bug, which I will investigate further and submit an issue or pull request for.

Accomplishments that I'm proud of

I spent a long time debugging and testing before I discovered the tungstenite library as the cause of the issue. I ended up resolving the issue by switching out the library for a more handwritten approach which accomplished the task required.

I'm proud of achieving the goal of the project I set out to make, as well as overcoming issues that I encountered along the way.

What I learned

I learned a lot more about WebAssembly over the course of the hack, as well as obtaining a better understanding of modular component based interface design.

What's next for tpp

The project could easily be expanded to work with other games, emulators and even PC games.

Other features that could be added are:

  • a voting system to reduce the chaos
  • a video stream into the web app (rather than relying on another display)
  • improved touch support

Built With

  • gba
  • retroarch
  • rust
  • webassembly
  • yew
Share this project: