Kinetic Swarm

Inspiration

I've always been fascinated by telekinesis in movies - Star Wars, that kind of thing. The idea of moving objects in the air just feels cool. When I started this project, I originally wanted to make a simple telekinesis experience in passthrough where you control a single object with your hands.

But then I thought - what if instead of one object, you controlled an entire swarm ?

I also made a deliberate design choice: every gesture requires both hands. No one-handed shortcuts. When you lift the swarm, both palms go up. When you freeze, both fists close. This two-handed requirement makes the experience feel more powerful and intentional - like you're actually channeling some kind of force, not just casually waving.

What it does

Kinetic Swarm is a WebXR hand tracking experience where you control 150 orbs using natural hand gestures. Each gesture triggers its own sound effect for immersive feedback.

Core Gestures:

Lift - Palms up to summon the swarm Freeze - Close fists to stop orbs mid-air Push - Thrust forward to blast orbs (directional aiming) Pull - Pull back to recall the swarm Explode - Spread hands quickly to scatter orbs Spin - Swipe to change orbit direction

Fine Control (Micro-gestures): Thumb Flick - Speed up spin Index Flick - Cycle vertical spread Middle Flick - Slow down spin

Shape & Tilt: Size - Hands apart = bigger orbit Y Tilt - Hand height difference tilts formation left/right Z Tilt - Hands forward/back tilts formation angle

Draw Mode: Enter - Touch thumbs together (hold 1s) Exit - Make fists (hold 1s) Orbs mirror your hand movements, letting you sculpt the swarm into custom shapes

Works in both Passthrough (AR) and Immersive VR modes.

How I built it

Built with Three.js for 3D rendering and the native WebXR Hand Tracking API. I wasn't aware of Meta's Interaction SDK at the time, so everything is pure WebXR.

I used AI assistance for programming - helping with debugging, refactoring, and exploring different approaches to gesture detection. The core design decisions, gesture vocabulary, and creative direction were all mine.

Hosted on GitHub Pages.

Challenges I ran into

Managing multiple gesture behaviors at once was tricky. Each orb needed to react differently depending on what gesture was happening - and gestures can interrupt each other.

Using a state machine for each orb solved this. Ground, lifting, orbiting, frozen, pushed, exploded, falling, drawing - each state handles its own logic. Clean separation, easy to debug, easy to add new behaviors.

Accomplishments that i'm proud of

The final result is really close to what I originally imagined. When I first pictured this project in my head - lifting a swarm of orbs with my hands, freezing them mid-air, pushing them away - the actual experience matches that vision, which is cool.

What i learned

WebXR is way more capable than I expected. Before this, I assumed VR development meant Unity, heavy SDKs, the whole process. Turns out you can build a full hand-tracking experience with just JavaScript and Three.js, running directly in the browser.

What's next for Kinetic Swarm

There's room to polish some interactions - a few gestures can still feel janky in certain conditions. Beyond that, I'm considering adding new gesture powers like wall or other effects.

But honestly, I might just leave it as is.