Boxing Orchestra

Inspiration

We took our inspiration from The Last Clockwinder VR, where you can automate tasks by recording your own movements and leaving behind “clones” that repeat them. The moment we saw that mechanic, it instantly clicked: this is exactly how modern music production works: layers, loops, automation, and timing. That connection sparked the idea for Boxing Orchestra - a VR experience where your own “ghosts” become your band. You hit drums, piano, or guitar once, walk away, and your clone keeps playing, building a live musical loop while you move around the space.

What it does

Boxing Orchestra turns your real-world movements into a fully automated musical performance. You hit an instrument once, drums, piano, or guitar, anything, and your ghost clone keeps looping that action with perfect timing. You can layer multiple loops, switch instruments, walk around the space, and build a full track entirely through movement. Need to stop a loop? Just punch your ghost in the face. Instant, physical, hilarious loop control.

Beyond the music use case, our core mechanic, recording embodied actions and spawning persistent autonomous clones, opens the door to much more: record and automate physical workflows, compare live motion to past “ghosts” for improving sport stats, or choreograph autonomous performers, animation helpers, and spatial design assistants.

How we built it

We built Boxing Orchestra on top of the Meta XR SDK, using its body tracking and spatial presence features to capture user movement inside VR. To create the “ghost automation” system, we record roughly 10 seconds of the player’s motion, capturing 1,000+ snapshots of their head, hands, and body transforms. These snapshots form a precise motion timeline that we replay as a fully animated ghost clone. Once spawned, each ghost runs its recorded loop indefinitely, perfectly repeating the user’s original rhythm and actions. The loop only ends when the player physically interacts with it, specifically, by punching the ghost in the face, which triggers its shutdown animation and despawns it from the scene.

Challenges we ran into

One of our biggest early challenges was the Unity 6 toolchain. While the new workflow was great, our version control completely broke down, Git refused to sync key project files, causing mismatched scenes, missing assets, and even rendering pipelines! We eventually solved it by rolling the entire project back to Unity 5, which restored stability and let us actually build instead of debug the editor.

The second major challenge was making the ghost mechanic feel good. Sequencing movement, sound triggers, and looping behavior in real time turned out to be way more complicated than expected. We experimented with different sampling rates, timelines, and interpolation methods before finding a setup that felt smooth, musical, and fun rather than chaotic.

Finally, we had to figure out how to turn all this tech into something anyone could enjoy, not just pro musicians. That meant simplifying interaction, making instruments responsive, and ensuring ghost loops always stayed in sync without users needing to think about timing.

Accomplishments that we're proud of

We’re genuinely proud of how far Boxing Orchestra came in such a short time. Not only did everything work, it actually worked better than we expected. Seeing the project evolve from an idea like (“what if you could punch your music loops?”) into a smooth, fully functional VR experience was honestly surprising. After spending most of the hackathon battling debugging issues, version control, and ghost playback quirks, we are really proud how it turned out.

What we learned

One big takeaway from this hackathon is that a tight time-constrained environment is not the ideal place to experiment with brand-new frameworks or unfamiliar engine versions. We learned that the hard way, when everything that could go wrong basically did. But we also learned something more important: even when the tools break, the versioning collapses, and the tech fights back, there’s always a way through. With enough debugging, teamwork, and stubbornness, every problem had a solution.

What's next for Boxing Orchestra

Honestly, the idea turned out so fun and surprisingly solid that we’re seriously considering polishing it up and publishing it. The foundation is already there, the ghost looping, the instruments, the interactions, so the next steps are actually pretty small compared to everything we already built.