Rocky: Unbeatable at RPS

Landing page
Entering the ring...
Before start
First loss
I keep losing
I give up
Nice try buddy

Inspiration

Honestly, we were inspired by two things: a funny YouTube video of someone playing rock paper scissors against a machine, and wanting an excuse to mess around with some new tech. Rocky gave us a reason to dive into hand landmark tracking, gesture classification, and GPIO output on a Raspberry Pi, all in service of the very serious mission of building a robot that never loses.

What it does

Rocky's job is simple: play rock paper scissors and never lose. 100% win rate always. Rocky is also deceivingly cute for how much of a pain it was to actually build.

How we built it

Mechanical build: Rocky truly is a hackathon build. You might have caught us cutting up empty Goldfish boxes with QNX merch we repurposed as an emergency X-Acto knife. We decorated him with stickers, the trading cards, basically anything we could turn into robot personality. It was funny and chaotic, but we still had to think through the actual mechanics, adding counterweights to the arms, making sure the movement wouldn't tip the whole body over, and figuring out where to put the circuitry so it stayed safe but was still easy to get to.

Gesture detection and classification: we run a live camera feed through MediaPipe Hands to pull out hand landmarks in real time, then use a rule based classifier on top of those points to figure out if you're throwing rock, paper, scissors, or nothing recognizable. To stop it from locking in too early on a shaky or half formed hand, we added a stability check that waits until the same gesture shows up a few frames in a row before trusting it.

Game state and decision logic: the Python backend keeps track of where each round is at, moving through waiting, countdown, armed, locked, and result. As soon as your gesture locks in, Rocky works out the move that beats it, updates the score, and pushes the current state out through a couple of simple HTTP endpoints so the UI always knows what's happening.

Hardware integration: on the Pi side, we hooked Rocky's software decisions up to real hardware using GPIO pins, servo motors, and a bunch of wiring. Whenever Rocky picks a move, the backend triggers the matching physical output, so the digital game state actually drives the robot's arm and gesture hardware in real life.

Real time web UI: the browser interface polls the backend for updates and keeps everything on screen in sync with the game loop, from the landing page and countdown through the camera stream, the lock animation, the result reveal, score changes, sound effects, and the final "give up" screen once someone's had enough.

Challenges we ran into

Honestly, the biggest challenge was just getting all the software and hardware pieces to actually cooperate. We ran into the usual mess of package, driver, and compatibility issues, especially working with QNX, which behaves nothing like what we were used to, so we needed extra debugging and random workarounds. On top of that, we had to carefully tune all the real time parts, the camera feed, gesture locking, countdown timing, sound effects, UI animations, and hardware outputs, so it all felt in sync instead of janky. Honestly most of the challenge wasn't any single piece, it was making all the separate systems feel like one smooth experience.

Accomplishments that we're proud of

Objectively, we're proud that Rocky actually works end to end: it can track a hand through the camera, classify a rock, paper, or scissors gesture, lock in the move, pick the winning counter, update the UI, play sounds, and talk to the Raspberry Pi hardware outputs.

But honestly, what we're most proud of is just how many times we had zero idea what we were doing and figured it out anyway. We actually started as a team of four, but two teammates had to bail early because of health emergencies, so it ended up just being the two of us scrambling. It was also the first time we'd ever worked together as a team, and it went way better than expected. We just split stuff up, trusted each other to handle our half, and kept pushing the project forward.

What we learned

Most importantly, we learned that diving into unfamiliar tools is way less scary when the project is funny enough to keep you motivated. We also learned how much polish actually comes down to tiny details, timing, sound, animation, and feedback all matter way more than you'd expect when you're trying to make something feel alive. Finally, we definitely learned to not underestimate the execution of a seemingly "simple" idea.

What's next for Rocky: Unbeatable at RPS

Since VAR's on everyone's mind with the World Cup going on, one idea is adding our own version so you can actually rewatc the exact moment Rocky caught and locked your gesture. We also want to save your history across rounds so Rocky starts picking up on your habits and can maybe call your move even earlier.

On the hardware side, the end goal is an actual robotic hand with servo motors that throws rock, paper, and scissors for real, not just on a screen. We didn't have enough on hand at the hackathon. We also want Rocky to have more personality, more voice lines, more reactions.

Another idea is rhythm matching. Instead of everyone being stuck on the same fixed countdown, Rocky could watch your hand motion with MediaPipe and match your pace, so the game feels more natural.