Hand Sabers

A webcam-powered rhythm game that turns your hands into virtual sabers; no VR headset or motion controllers required.

Comment

Inspiration

I wanted to know if a rhythm game could feel physical and immersive without a VR headset or motion controllers.

Almost everyone has a webcam, so I started experimenting with real-time hand tracking, mapping each hand to a virtual saber. That small technical experiment slowly turned into Hand Sabers.

What it does

Hand Sabers is a rhythm game you play in your browser, using just your hands.

Your webcam tracks both hands and turns their movement into two virtual sabers in a 3D scene. Cubes fly toward you on different lanes, and you have to hit each one with the right hand, at the right time, in the right direction.

Right now the project includes:

  • real-time tracking of both hands
  • scoring, accuracy, misses, and combos
  • playable and importable maps
  • a map library with leaderboards
  • a timeline-based map editor
  • one-hand mode and other gameplay settings
  • multiplayer rooms and lobbies
  • remote tracking through a paired phone
  • music-reactive visual effects
  • performance diagnostics and automated checks

How I built it

Hand Sabers runs on TypeScript, Three.js, Vite, Node.js, Express, WebSockets, and browser-based computer vision.

The project already existed before OpenAI Build Week. During the official submission period, I used Codex with GPT-5.6 to add new features and improve existing systems.

During Build Week, I worked on:

  • remote hand tracking from a paired phone
  • authenticated tracking connections with packet validation
  • phone-based calibration and tracking controls
  • multiplayer host rules, lobby map selection, and chat
  • synced previews of remote players' hands
  • a 3D preview inside the map editor
  • beat-reactive tunnel rings and scene effects
  • configurable graphics and music-effect settings
  • performance profiling for frame processing
  • better error isolation and runtime reliability
  • refactoring across gameplay, networking, diagnostics, and the editor

The Git history keeps this work clearly separated from the earlier version of the project. This submission only covers what I built during the official Build Week period.

I used Codex as a coding partner. I gave it specific tasks, had it look through the existing code, reviewed what it proposed, and tested everything myself.

It was especially useful for tracking down related code across the project, building small contained features, refactoring bigger modules, hunting down bugs, and making sure changes didn't break anything else.

All the product decisions, feature choices, code reviews, and gameplay testing were still on me.

Challenges I ran into

The hardest part was turning noisy webcam input into movement that actually feels stable and responsive.

Hand tracking shifts with lighting, camera angle, fast movement, and moments when a hand briefly disappears from view. On top of that, raw coordinates carry small jitters that make the sabers shake.

Too little smoothing and the controls feel unstable. Too much and you get noticeable lag. Getting the balance right took a lot of hands-on testing — automated checks alone couldn't tell me if it felt good.

Keeping the codebase manageable as it grew was another challenge. Gameplay, multiplayer networking, phone tracking, the editor, and the visual effects all touch each other.

During Build Week, I broke several large systems into smaller modules for transport, protocol validation, map timing, spawning, hit feedback, diagnostics, and the editor.

Accomplishments I'm proud of

I'm proud of how far Hand Sabers has come — from a hand-tracking experiment to a playable rhythm game with its own map editor, multiplayer, remote phone input, visual effects, and performance tools.

The best moment is still watching someone raise their hands, see the sabers follow, and immediately get how to play — no explanation needed.

No headset, no controllers. Just a webcam and a bit of space to move.

What I learned

AI coding tools work best when the task is clearly defined and the developer still understands, reviews, and tests what comes out.

Codex helped me move through a large codebase faster, especially for finding connected systems, handling repetitive refactoring, and chasing down errors.

I also learned that automated tests can't tell you whether a rhythm game feels responsive or fun. That still takes real gameplay testing and your own judgment.

What's next for Hand Sabers

This is still a preview build, and I want to keep improving it. Next up:

  • better tracking stability and lower latency
  • more maps and gameplay modes
  • easier installation and deployment
  • broader multiplayer testing
  • support for more devices
  • more work on the map editor

About the builder

I'm Michał Oślizło, also known online as VermiNew — a programming enthusiast from Podkarpacie, Poland.

I focus on web and app development, and I've passed the INF.03 and INF.04 qualifications. I like picking up new technologies by putting them to work in real projects and seeing how different systems fit together.

Built With

Share this project:

Updates