FingerFlow! Rhythm Line

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Inspiration

FingerFlow began with a simple question: could a rhythm game feel like conducting music with one finger?

I already had a working rhythm game controlled by taps. I wanted to bring that mechanic into spatial computing while keeping its simplicity. Instead of tapping a screen, players would guide the line with short, deliberate gestures of their index finger and hand.

AR became central to the idea. A compact musical track could appear in your surroundings, without needing to fill your entire view. An immersive mode could then expand the same game into a complete neon world.

What it does

FingerFlow is a hand-tracked rhythm game in which a neon line moves automatically through a musical course. Players turn at the right moment by alternating gestures: left, right, left, right.

The current version includes:

• Passthrough mode, with a compact track, gems and restrained visual effects.
• Immersive mode, with reactive scenery, neon lighting and virtual LED gloves.
• 32 songs across five categories and 29 playable characters.
• An interactive tutorial with guided turns and short rewinds when players miss.
• Audio calibration using ten taps synchronized to audible beeps.
• Local progress tracking and per-song demo leaderboards. Players can navigate the menus with their hands and pause by closing their playing hand.

How we built it

I adapted my existing browser game using Three.js, WebXR and the Web Audio API.

Reusing the original simulation, music timing and visual assets let me focus on the spatial experience: hand input, comfortable placement, readable menus and the difference between passthrough and immersion.

The gameplay follows the audio clock. Hand-joint tracking feeds a custom gesture recognizer that detects deliberate lateral movements while filtering small tremors and rebounds. Menu interaction uses the platform’s native pointing ray and pinch selection.

I used AI-assisted coding during development, alongside AI-generated branding and song artwork, then refined the results through repeated testing on Meta Quest 2.

Challenges we ran into

The hardest challenge was making a gesture count exactly once.

A deliberate flick often ends with a small involuntary movement in the opposite direction. Early versions sometimes interpreted that rebound as another turn. Other versions filtered too aggressively and missed intentional gestures.

Recording motion and timing data helped me investigate those failures and refine the balance between responsiveness and stability.

Another challenge was visual comfort. Effects that looked good on a monitor could obscure the track or make the spatial experience feel unstable. I reduced unnecessary effects in passthrough, separated the presentation of AR and VR, and optimized animated panels.

The tutorial also needed iteration. Rather than restarting a whole practice sequence after a mistake, it now rewinds only the missed section and preserves successful turns.

Accomplishments that we're proud of

Turning an existing tap-controlled rhythm game into a playable, controller-free XR experience while preserving its core mechanic.

Our biggest achievement was making the gestures feel reliable. Through repeated testing on Meta Quest 2, we refined the recognition to distinguish intentional flicks from small tremors and involuntary rebounds.

We’re also proud of giving AR and VR their own presentation: a compact, readable track in passthrough and a richer neon world in immersive mode.

Finally, the guided tutorial, short rewinds and ten-tap audio calibration make the game easier to learn and personalize. Together with 32 songs and 29 playable characters, these features helped FingerFlow grow from an interaction experiment into a complete playable experience.

What we learned

A simple gesture requires careful interaction design.

Players need clear feedback, predictable timing and an easy way to pause. Menus need to follow familiar platform conventions, while the gameplay can introduce its own movement vocabulary.

I also learned that AR and VR benefit from different presentations. In AR, clarity and restraint help the game fit into the room. In VR, the same mechanic can support a richer visual environment.

What's next for FingerFlow! Rhythm Line

The playable WebXR version has been tested on Meta Quest 2. My next goal is to bring FingerFlow to a packaged Meta experience and explore adaptations for Meta’s new glasses, validating the controls and presentation against each device’s capabilities.

I also plan to replace the clearly labeled demo rankings with a real leaderboard service and continue refining accessibility, comfort and performance.

Built With

• css3
• html5
• indexeddb
• javascript
• three.js
• webaudioapi
• webxr