MIRL - MINECRAFT IRL

Play Minecraft with your body. Swing a real sword, raise a shield, fire a bow, sprint, jump, and look around IRL while IMU sensors and computer vision translate every move into the game.

Comment 3

Inspiration

Minecraft is one of the most physical games ever made. You sprint, jump, swing swords, raise shields, and draw bows, yet we play it hunched over a keyboard. We wanted to close that gap. What if playing Minecraft actually felt like being in Minecraft? So we built the game's most iconic items out of cardboard, strapped sensors to them, and turned our own bodies into the controller.

What it does

Minecraft IRL replaces your keyboard and mouse with your body and a set of handheld props:

  • Auto-equip: physically pick up a prop and the game instantly switches to it. Grab the sword and it's selected in your hotbar and in your hand, grab the bow and you're ready to shoot
  • Sword: swing the cardboard sword in real life and your character attacks in-game
  • Shield: raise the shield and your character blocks
  • Bow: draw and release the bow to fire an arrow Each prop houses an ESP32 with an IMU that streams motion data to a laptop, where gesture detection maps real movements to game inputs in real time. On top of the props, a computer vision layer tracks your body through a webcam:
  • Run in place (pumping your arms in a running motion) and your character sprints
  • Jump in real life and your character jumps in-game
  • Turn your head and the in-game POV follows. Facial pose estimation controls the camera, so looking around IRL is looking around in Minecraft Everything works simultaneously: you can sprint toward a mob, jump, turn to face it, grab your sword, and swing, all with your body, no keyboard, no mouse.

How we built it

  • Props: Sword, shield, and bow built from cardboard during the hackathon, each housing an ESP32 dev board with an onboard IMU
  • Motion detection: IMU accelerometer/gyro data streamed from each ESP32 to a laptop, where our gesture classifiers distinguish a sword swing, a shield raise, and a bow draw-and-release
  • Auto-equip: The system detects when a prop is picked up and automatically selects the matching item in the in-game hotbar, so switching weapons in real life switches them in-game
  • Computer vision: OpenCV pipeline handling three tracks: arm-motion detection for running, jump detection, and facial pose estimation for head-tracked camera control
  • Game input: Detected gestures and poses translated into synthetic keyboard/mouse events driving an unmodified copy of Minecraft

Challenges we ran into

  • Gesture vs. noise: A sword swing and an excited hand wave look similar to an IMU, so we tuned thresholds and added debouncing to stop the game from spamming attacks
  • Pickup detection: Telling the difference between picking up a prop (equip it) and just bumping the table it's sitting on (do nothing)
  • Smooth head tracking: Pose estimation had to be filtered enough to control the camera without making the POV jittery or nauseating
  • Real-time performance: Running multiple CV tracks (run detection, jump detection, face pose) alongside live gesture classification without dropping frames
  • Latency: Body-to-game input has to feel instant or the whole illusion breaks
  • Cardboard engineering: Building props sturdy enough to survive repeated full-force demo swings, out of cardboard, in one weekend

Accomplishments that we're proud of

  • A fully working end-to-end system: three sensored props plus full-body CV controlling live Minecraft, built entirely within the hackathon
  • Seamless weapon swapping: put down the sword, grab the bow, and the game just follows along
  • The head-tracked POV. Turning your head and having the game follow is the moment that sells the whole experience
  • Fusing multiple simultaneous input streams (IMU gestures + CV body tracking + face pose) into one coherent control scheme

What we learned

  • Real-time gesture classification from raw IMU data, and how much filtering and thresholding it takes to feel reliable
  • Building a multi-track computer vision pipeline with OpenCV and pose estimation that runs fast enough for live gameplay
  • ESP32-to-laptop data pipelines and translating sensor events into low-latency game input
  • Rapid physical prototyping: cardboard engineering is a real skill

What's next for Minecraft IRL

  • Haptic feedback in the props (rumble on hit, tension on the bow)
  • More items: pickaxe for mining, fishing rod, eating gestures
  • Full skeletal pose tracking for crouching, swimming, and placing blocks
  • Cutting the laptop out entirely for a fully standalone setup

Built With

+ 4 more
Share this project:

Updates