Fingerspell

Fingerspell turns ASL fingerspelling into text, entirely on a tiny chip. A camera watches your hand, an Ethos-U85 NPU reads each letter, and the words appear on screen, with no cloud and no phone.

Comment

Inspiration

ASL has signs for most words, but names, addresses, medications and anything new get fingerspelled, one letter at a time on one hand. That is exactly the content a Deaf person most needs to get across at a pharmacy counter or a front desk, and exactly what a hearing person who does not sign cannot follow. I wanted a small device that reads fingerspelling and puts the letters on a screen, with nothing leaving the device: no phone, no cloud, no account. The Alif Ensemble E8 with its Ethos-U85 NPU looked like enough compute to run real hand models at camera rate, so Fingerspell became my hackathon project.

What it does

Point the DevKit's camera at a signing hand. The board finds the hand, extracts its 21 keypoints, classifies the handshape into a letter, and builds the letters into words on the LCD. All 26 letters work: 24 still shapes from the classifier, and Z and J from a motion rule (hold D or I, then draw the stroke). The panel shows the live hand box with the skeleton drawn on it, the top letter with its confidence bar, the next two guesses, the committed text, and the per-stage timings. The joystick deletes a letter or clears the line. The board's accelerometer measures gravity so the hand is levelled against true vertical, and tilting the board does not change the letter. It runs at 15 to 16 fps with a hand in view, entirely on the board.

How We built it

The firmware is a CMSIS-Toolbox solution on top of the Arm ExecuTorch example. Two MediaPipe models run on the Ethos-U85 as two methods of one ExecuTorch program: BlazePalm (palm detection, 192×192) and hand_landmark_lite (21 landmarks from a 224×224 crop). I converted both from TFLite through ONNX into PyTorch, quantized them int8 with the Arm TOSA quantizer and compiled them with Vela. Every operator in both models runs on the NPU.

The Cortex-M55 does everything that is geometry or rules: the 320→192 resize, decoding the detector's anchors, rotating and cropping the hand so the landmark model always sees fingers up, levelling the landmarks with the IMU, a 63→128→24 MLP that maps keypoints to letters, and a decoder with commit, hold, release and space rules plus the Z/J motion rule. The MLP is trained in PyTorch in minutes on landmarks from the Marxulia dataset plus landmarks extracted through the board's own pipeline, and its weights are emitted straight into a C header.

Around that sit an MT9M114 camera stack streaming RGB565 by DMA into SRAM, the CDC200 display path to the 4.3" panel, the ICM-42670-P accelerometer over I3C, and the joystick on GPIO. Every C stage is pure and tested on the host against a numpy reference; the decoder alone has 352 checks. A telemetry block readable over J-Link lets me watch the live loop even when the UART is down.

Challenges We ran into

  • Pixels did not transfer. My first classifier (MobileNetV2) scored 99 % on its dataset and 0.3 to 0.4 confidence on a real hand in a lit room. Switching the letter decision to keypoints fixed that by construction: the classifier never sees a pixel.
  • An input range cost half the detections. BlazePalm takes [0, 1], not [-1, 1]. The wrong range put boxes off the hands (31 % hit rate vs 76 %). Hours of measuring against MediaPipe ground truth to find one constant.
  • The quantize ops ate the frame budget. ExecuTorch's portable quantize/dequantize at the method boundary cost 25 ms per frame on the M55. Quantizing the IO so the firmware writes and reads int8 removed them.
  • Memory. 16-bit activations wanted 2.3 MB of scratch that SRAM0 cannot give, so everything is a8w8. The camera ring overwrote the A32 boot stub, NPU buffers in SRAM1 were unsafe, and a warm reset ran stale cache lines. Three board-level fixes.
  • Things that only happen on hardware. The cycle counter only runs with a debugger attached, so timings printed 0.0 ms until I switched to the SoC's reference clock. The UART died after a USB reconnect, which is why telemetry over J-Link exists.
  • Real hands. The landmark model's handedness flipped on crossed shapes like R, small hands lost presence, camera tilt moved letters, and noisy frames spammed "B B B B". Fixed handedness, a presence hold, gravity levelling and three stages of temporal averaging came out of the signing sessions.

Accomplishments that We're proud of

Two converted vision models and a custom classifier running in one program with every operator on the NPU, at 15 to 16 fps, in 2.6 MB of MRAM. A letter classifier that holds 94.5 % on held-out landmarks from the board's own pipeline. Motion letters without a sequence model. A decoder that commits once per hold and never doubles a letter. And a device I can hand to someone and have it spell a word with no setup.

What We learned

Measure on the board, not in the simulator: both NPU models run several times slower than Vela's estimate, and the CPU-side costs (resize, quantize ops, crop) were the real bottlenecks at first. Keypoints beat pixels whenever the deployment domain differs from the training photos. Quantization boundaries, memory maps and cache state are design decisions, not details. And instrument everything early: the telemetry block and host tests paid for themselves many times over.

What's next for Fingerspell

  • More keypoints captured from the board per letter to retrain the MLP on the confusions it still has (Q/G, F/I, R/X, M/N).
  • Close the NPU speed gap against Vela's estimate, which would lift the frame rate for both models.
  • Word-level suggestions on top of the letter stream, and a two-way mode with a microphone path so the hearing person can answer (plans drafted, not built).
  • Power measurement, and a port to the smaller AppKit so it fits in a pocket.

Built With

Share this project:

Updates