Hand gesture recognition

• 

  pinch_demo

• 

  thumbs_up

• 

  no_gesture

• 

  palm

Inspiration

Most gesture-recognition demos either lean on cloud inference trading away privacy and latency or stop at a single accuracy number with no real story about how the model actually performs on the hardware it's meant to run on. I wanted to build something that lived entirely on-device: a hand could control a phone in real time, with zero video ever leaving it, and I wanted to be honest and rigorous about what it actually costs to get there — model size, latency, and where Arm-specific optimization genuinely helps versus where it doesn't.

What it does

The app watches the phone's camera, and when it detects meaningful motion, it extracts 21 hand landmarks with MediaPipe and classifies the pose in real time using two lightweight neural networks running entirely on the phone's Arm CPU:

• A static MLP recognizes discrete poses — thumbs up, open palm, fist, and a pinch — from a single frame.
• A temporal TCN recognizes motion gestures — swipe left, swipe right, and a wave — from a rolling 20-frame window of landmarks.

A cheap frame-differencing motion gate sits in front of both models, so the expensive landmark-extraction and classification stages only run when something is actually happening in frame — the system idles for free otherwise.

How I built it

I started by prototyping entirely with a laptop webcam and MediaPipe in Python, recording hand-landmark sequences and training two small PyTorch models (an MLP for static poses, a 1D TCN for dynamic gestures) on normalized, wrist-relative keypoint features plus engineered fingertip- distance features. From there I ran a full optimization pipeline: FP32 baseline → INT8 post-training quantization → export via ExecuTorch with the XNNPACK backend, targeting Arm CPU kernels. I measured accuracy, model size, and latency at every stage rather than only reporting a final number.

On the Android side, I built the live app in Kotlin with Jetpack Compose and CameraX for the camera pipeline, MediaPipe's Android HandLandmarker for on-device keypoint extraction, and ExecuTorch's Android runtime to load and run my exported .pte models directly on the phone — with live latency instrumentation for every stage (motion gate, landmark extraction, classification) displayed on screen during testing.

Challenges I ran into

• Feature scaling mismatch: my models were trained on StandardScaler- normalized features, but my first on-device build fed raw features straight to the model — silently collapsing predictions into one or two classes. I fixed this by baking the scaler directly into the exported model as a buffer, so the .pte file normalizes internally.
• Windows/ExecuTorch export tooling: the flatc FlatBuffers compiler ExecuTorch needs for export isn't reliably bundled for Windows, so I moved the export step to Google Colab, which runs on Linux and matches ExecuTorch's actual test environment.
• Camera-frame to bitmap conversion: raw camera buffers have row padding and sensor-specific rotation that don't match what's shown on screen — I had to handle stride correctly and work out the right rotation/mirror transform specific to my device through direct landmark-coordinate logging rather than guessing visually.
• Class confusion in early training: fist and pinch were frequently confused with palm, traced back to self-occlusion in MediaPipe's landmark estimation for closed/curled poses partly mitigated by making the poses themselves more geometrically distinct and adding fingertip-distance features.
• Domain shift: training data collected on a laptop webcam doesn't perfectly transfer to a phone's front camera — my dynamic gestures show this most clearly, with swipe_left working reliably while swipe_right and wave need more phone-captured training data.

Accomplishments that I am proud of

Getting a complete, honest, end-to-end optimization pipeline working and measured at every stage — not just a demo, but real before/after numbers for FP32 → INT8 → ExecuTorch/XNNPACK, plus a working adaptive motion-gating system that measurably reduces unnecessary computation. I am also proud that when things broke — and a lot did — I debugged with real data (Logcat landmark logging, confusion matrices, stage-by-stage latency) rather than guessing.

What I learned

INT8 quantization is not automatically a latency win — on models this small, plain PyTorch CPU inference showed quantize/dequantize overhead outweighing any compute savings, and it took Arm-optimized XNNPACK kernels to actually realize a benefit. I also learned how much of the total pipeline cost lives in hand-landmark extraction rather than classification itself, which reframes where the real optimization opportunity is: not shrinking the classifier further, but running it less often — exactly what the motion gate is for.

What's next for Hand gesture recognition

• Collect training data directly on-device to close the phone-camera domain gap and fix swipe_right/wave reliability
• Run the full three-configuration benchmark (FP32-always-on vs. INT8-always- on vs. full-optimized-with-gating) including a sustained thermal test, to quantify real battery and thermal-throttling benefits of adaptive gating
• Explore per-finger curl angle features to further separate fist/pinch/palm
• Map gestures to real system actions (media control, navigation) instead of just an on-screen label, for a more complete end-to-end demo

Built With

• android
• android-studio
• arm
• camerax
• computer-vision
• edge-ai
• executorch
• gesture-recognition
• google-colab
• jetpack-compose
• kotlin
• mediapipe
• mlp
• mobile-ai
• on-device-ml
• python
• pytorch
• quantization
• tcn
• tflite-alternative
• xnnpack