Flick Note

The final hardware setup
Reading data sent through bluetooth
Gameplay UI
esp32 + IMU + glove (before sewing)

Inspiration

As the name suggests, Flick Note is inspired by the flick notes in Project Sekai (the notes you hit by swipe instead of tapping). They're the most satisfying part of the game, but on a phone or tablet they're still just a finger sliding across glass. We wanted a physical version: a glove on your hand, where hitting a note means snapping your whole hand toward it, instead of the traditional taps.

Affordable IMU's measure hand movements are rotation extremely well and with almost no delay, which makes them a natural fit for wrist flicks. We set out to build a controller that you wear instead of touch, and a rhythm game designed around exactly what this sensor is good at.

What it does

Flick Note is a rhythm game played with an IMU glove.

A small microcontroller on the wrist streams motion data to the PC over Bluetooth Low Energy in 10 millisecond intervals.
On screen, a glowing virtual glove mirrors your real hand in real time, and a laser pointer shows exactly where you're pointing.
Notes fly toward you down four lanes (up, down, left, right). To hit one, snap your hand toward its lane so the laser is on the note's ring as it arrives, like a flick note played with your whole hand.
Timing: Perfect if you're on the lane when the note reaches the ring, Good if you got there within ±140 ms, otherwise Miss. A shrinking approach circle shows the exact moment each note lands, combos build a score multiplier up to ×4, and each run ends with a letter grade (S to D).
A quick wrist flick starts the song, and snaps the virtual hand into a fist.
It's designed to run untethered: the glove needs only power (a small battery or USB power bank), and the game reconnects on its own if Bluetooth drops.

How we built it

Hardware: an Adafruit Feather ESP32-S3 with an ICM-20948 9-axis IMU (accelerometer, gyroscope, magnetometer) connected over STEMMA QT / I²C.

Firmware (C, ESP-IDF on FreeRTOS, built with PlatformIO): reads the accelerometer, gyroscope, magnetometer and temperature at 100 Hz and broadcasts each reading as a compact binary packet over a custom BLE GATT service (NimBLE stack). It keeps advertising even while connected, so a crashed PC app can always reconnect.

Python bridge: connects to the glove with bleak, calibrates the gyro while the hand is still, and fuses gyro + accelerometer (+ magnetometer, if calibrated) with a Madgwick filter into a smooth 3D orientation. The core of the filter integrates the gyro's angular velocity as a quaternion and nudges it toward gravity:

$$\dot{q} = \tfrac{1}{2}\, q \otimes (0, \omega) \;-\; \beta\, \frac{\nabla f}{\lVert \nabla f \rVert}$$

Aiming: the laser points along the hand's forward axis, $\mathbf{f} = R(q)\,\hat{\mathbf{x}}$ (the fingers lie along the sensor's $x$ axis). You're on a lane when the angle between that direction and the lane is under 14°:

$$\theta = \arccos\big(\mathbf{f} \cdot \mathbf{d}_{\text{lane}}\big) < 14^\circ$$

The lanes sit 22° and 28° from center, so pointing straight ahead aims at nothing, and every hit takes a real snap of the hand toward the note.

Gestures: a wrist flick is a spike in rotation speed above 350 °/s, and shakes and flips come from the accelerometer. The bridge also converts from the sensor's right-handed axes to Unity's left-handed, Y-up axes, and sends everything (orientation plus gesture events) to the game as JSON over local UDP.

Game (Unity 6, C#): receives the stream, drives the virtual hand (the IMU gives rotation; an "arm model" with a virtual shoulder and elbow turns that into a believable hand position), and runs the rhythm game: charting, timing windows, scoring, a neon-glow look built from additive blending, a procedurally generated backing track (no audio files), and a jointed glove model whose fingers curl on each flick.

We also built a live Python dashboard (plots of all sensors, CSV recording, compass calibration) that we used constantly while developing, and a self-test mode in the game where a bot plays the full song and saves screenshots, so we could check every build automatically.

Challenges we ran into

Position from an IMU is (almost) impossible. Our first goal was tracking where the hand is, but position means integrating acceleration twice, so any small sensor bias $b$ grows quadratically: $$e(t) = \tfrac{1}{2}\, b\, t^2$$ A bias of just $0.05\ \text{m/s}^2$ gives $2.5\ \text{m}$ of error after only 10 seconds. We built a position tracker with zero-velocity updates that works for short move-and-pause motions, but ultimately redesigned the game around what the IMU measures well: orientation.
Coordinate systems. The sensor uses right-handed axes and Unity uses left-handed ones, so rotations flip direction, not just axes. We derived the conversion and verified it numerically against matrix math before trusting it.
Finding the right "flick". We tried punches, then "aim at the lane and flick", then flicks judged by their direction. Every version was too hard to play with a glove over Bluetooth: the flick itself swings your hand off the target, and the Bluetooth timing jitter ate into the timing windows. Playtesting led us to the version that felt best: the snap of your hand toward the lane is the flick.
Bluetooth on Windows could keep a dead connection alive after an app crashed, hiding the glove from new scans. We fixed it in firmware by advertising even while connected.
Unity build surprises: shaders stripped from builds (everything rendered magenta), UI text corrupting because font sizes changed every frame, and a Unity install that kept failing until we moved everything onto a drive with enough space.

Accomplishments that we're proud of

A complete wearable pipeline: custom firmware, then Bluetooth, sensor fusion, and a polished game, all working end to end.
Reliable streaming: 100 samples per second over BLE with no dropped samples in our tests, and roughly 30 ms of Bluetooth delay.
Orientation that matched the true orientation to within about 3° in our filter tests, fast enough that the virtual hand feels 1:1.
A distinctive dark-red neon look with a fully jointed virtual glove, built entirely from code-generated shapes, with no 3D modelling or asset store.
An automated test bot that plays the whole song and captures screenshots, which caught several bugs that never showed up in the code itself.

What we learned

Sensor fusion in practice: why gyros drift, why accelerometers are noisy, and how a Madgwick filter blends the two; plus magnetometer hard-iron calibration.
What an IMU can and can't do: rotation is excellent, but position isn't reliable, and designing around a sensor's limits beats fighting them.
Embedded + BLE: ESP-IDF, NimBLE GATT services, packet design and connection handling.
Game feel: latency compensation, timing windows, and visual feedback matter as much as raw accuracy, and playtesting beats theory.

What's next for Flick Note

Haptics: a small vibration motor so you feel every beat and hit.
Finger tracking with flex sensors, for grab, fist and point gestures.
Real songs: load your own music with auto-generated or hand-made charts.
More note types that use the glove's strengths: directional wrist-flick notes (with a higher-range gyroscope and wider timing windows), wrist-twist notes and hold-and-follow notes (like Project Sekai's slides).
Two gloves and multiplayer: one cheap board per player instead of a headset per player.
Surface drumming mode: tap any real surface in rhythm, using the accelerometer's sharp impact spike for precise timing and real tactile feedback.