MIRL - MINECRAFT IRL

Play Minecraft with your body. Swing a real sword, raise a shield, fire a bow, sprint, jump, and look around IRL while IMU sensors and computer vision translate every move into the game.

Comment 5

https://youtu.be/vkHMMlXg5PI

GitHub: https://github.com/Solaror0/HT62026-MIRL

💡 Inspiration

Minecraft is one of the most physical games ever made. You sprint, jump, swing swords, raise shields, and draw bows, yet we play it hunched over a keyboard. We wanted to close that gap. What if playing Minecraft actually felt like being in Minecraft? So we built the game's most iconic items out of cardboard, strapped sensors to them, and turned our own bodies into the controller.

🎮 What it does

Minecraft IRL replaces your keyboard and mouse with your body and a set of handheld props:

  • Auto-equip: physically pick up a prop and the game instantly switches to it. Grab the sword and it's selected in your hotbar, put it down and your previous slot is restored
  • Sword: swing the cardboard sword in real life and your character attacks in-game
  • Shield: raise the shield and your character blocks
  • Bow: draw and release the bow to fire an arrow

On top of the props, a computer vision layer tracks your body through a webcam:

  • Run in place (pumping your arms) and your character moves, cross a higher threshold and you sprint
  • Jump in real life and your character jumps in-game
  • Turn your head and the in-game POV follows, so looking around IRL is looking around in Minecraft

Everything works simultaneously: you can sprint toward a mob, jump, turn to face it, grab your sword, and swing, all with your body. No keyboard, no mouse, and no synthetic keypresses either: every action drives the Minecraft client natively through a custom Fabric mod we wrote at the hackathon.

🛠️ How we built it

The whole system is three layers speaking one shared protocol.

Hardware layer. Each cardboard prop (sword, shield, bow) houses an ESP32 with an MPU6050 IMU. Gesture detection runs directly on the firmware: acceleration magnitude thresholds for sword swings, orientation for shield raises, draw-and-release sequencing for the bow. Each prop broadcasts compact state packets over ESP-NOW to a master ESP32, which forwards them to the laptop over USB serial as a one-line CSV protocol: node_id,picked_up,left_click,right_click.

Computer vision layer. A Python client built on OpenCV and MediaPipe Pose Landmarker watches the webcam and extracts three body signals from a single pose model call per frame: head turn (nose position against a calibrated neutral point), run/sprint (arm-pumping motion with two thresholds), and jump (vertical velocity spikes against a rolling baseline). Detections stream to the mod over a local TCP socket using the exact same CSV protocol as the hardware.

Game layer. A custom Fabric mod for Minecraft 1.21.1 (Java 21) receives both streams. A dedicated serial thread and a TCP server feed packets into one unified pipeline: parser, per-node handlers, and an action queue that's drained once per client tick, the only place game state is ever mutated. Rising-edge detection guarantees one real swing equals exactly one in-game attack. Because hardware and CV speak the same protocol, nothing downstream cares where an input came from, and a debug simulator let us build the entire game layer with keybinds before the hardware even existed.

🧗 Challenges we ran into

  • Gesture vs. noise: A sword swing and an excited hand wave look similar to an IMU, so we tuned thresholds on the firmware and added rising-edge detection in the mod to stop the game from spamming attacks
  • Pickup detection: Telling the difference between picking up a prop (equip it) and just bumping the table it's sitting on (do nothing)
  • Smooth head tracking: Pose estimation had to be filtered and calibrated against a neutral point to control the camera without making the POV jittery or nauseating
  • Threading: Serial input, TCP input, and Minecraft's client state all live on different threads. We solved it with an action queue that bridges everything onto the client tick thread
  • Latency: Body-to-game input has to feel instant or the whole illusion breaks
  • Cardboard engineering: Building props sturdy enough to survive repeated full-force demo swings, out of cardboard, in one weekend

🏆 Accomplishments that we're proud of

  • A fully working end-to-end system: three sensored props plus full-body CV controlling live Minecraft through our own Fabric mod, built entirely within the hackathon
  • One unified wire protocol from ESP-NOW mesh to webcam to game, so every input source flows through the same pipeline
  • Seamless weapon swapping: put down the sword, grab the bow, and the game just follows along, restoring your hotbar when you're done
  • The head-tracked POV. Turning your head and having the game follow is the moment that sells the whole experience

📚 What we learned

  • Writing a Minecraft Fabric mod, including the client threading model and how to safely mutate game state from external input streams
  • Real-time gesture classification from raw IMU data, and how much filtering and thresholding it takes to feel reliable
  • ESP-NOW wireless mesh networking between microcontrollers and serial protocol design
  • Building a real-time pose estimation pipeline with OpenCV and MediaPipe fast enough for live gameplay
  • Rapid physical prototyping: cardboard engineering is a real skill

🚀 What's next for Minecraft IRL

  • Haptic feedback in the props (rumble on hit, tension on the bow)
  • More items: pickaxe for mining, fishing rod, eating gestures
  • Full skeletal pose tracking for crouching, swimming, and placing blocks
  • Cutting the laptop out entirely for a fully standalone setup

Built With

+ 20 more
Share this project:

Updates