MaToS

Web features
Github repository
Demo testing
Web interface

Inspiration

[Personalize this.] Fingerspelling is how signers spell names, places and any word without a sign, but turning it into text usually needs special hardware or a GPU. We wanted MaToS to work from an ordinary webcam on a laptop CPU.

What it does

MaToS reads fingerspelled ASL from a webcam and turns it into text live. Signing a word letter by letter produces a word, a "space" gesture ends it, and a pause ends the sentence. A spelling corrector then repairs the classifier's mistakes, and a small T5 model turns the gloss into English:

MY NIME ISSS RAFDY → "My name is RAFDY."

How we built it

Pipeline (all on CPU):

Hand tracking: MediaPipe HandLandmarker gives 21 3D landmarks per frame.
Features: landmarks are normalized (wrist at the origin, scaled by palm size) and turned into geometric features (distances, joint angles, palm-normal projections). These target look-alike letters (M/N/T, U/V/R, A/E/S, K/P, G/Q).
Classifier: an MLP (28 classes: A–Z, space, nothing) trained with mirror, rotation, scale and jitter augmentation. It is exported to ONNX and runs in ONNX Runtime or OpenCV DNN.
Verifier: a tiny logistic regression on top-1 probability, margin and entropy estimates whether the prediction is right.
Decoder: time-based (fps independent). It smooths probabilities, requires a letter to be held before typing it, and uses a space gesture, dips or long holds for double letters, and pauses for word and sentence ends.
Autocorrect: a custom weighted edit distance over a 7,500-word dictionary plus personal names, with cheaper costs for dropped letters and look-alike pairs. A beam search joins the per-word winners.
T5-small polish: a fine-tuned gloss→English model on ONNX, followed by a validator that falls back to the corrected gloss if T5 loses a name or rambles.

Training: three public ASL alphabet datasets plus our own captures. Every choice (calibration, verifier, hyperparameters) comes from leave-one-source-out predictions, so a model is always scored on a signer and camera it never saw. A bundle is only exported if ONNX Runtime reproduces the trained model's probabilities.

Challenges we ran into

Look-alike letters: better geometric features helped more than a bigger network.
Honest accuracy: random splits of one dataset overstate accuracy because the same signer and background appear in train and test, so we switched to leave-one-source-out.
J and Z: they are motion letters, a per-frame classifier sees only one pose, and one public dataset has no J/Z at all. We leaned on dictionary coverage and autocorrect to recover them.
Double letters vs. flicker: a held or shaky hand can retype a letter, while English has real doubles like HELLO. We built dedicated repeated-letter handling.
Autocorrect mangling names: LOKY kept turning into LOOK. The raw chunk now competes with dictionary words, and a personal names list gets a high prior.
Deployment: OpenCV DNN can't run tree models, so we kept the MLP exportable as plain ONNX ops. Training needs only numpy, and we replaced a large LM reranker with a small T5.
T5 hallucination: a validator checks its output against the input.

Accomplishments that we're proud of

Real-time fingerspelling to text on a CPU with a plain webcam
It can run on cloud systems
Evaluation on unseen signers:

[loso] cross-source accuracy 0.978 (worst source 0.977) | temperature 0.71 (NLL 0.109 -> 0.079) | verifier AUC 0.9828824188055334 [loso] most confused: nothing->space 43%, nothing->C 43%, N->M 13%, nothing->Q 7%, D->O 2%, U->R 1%, M->nothing 1%, X->nothing 1% [export] parity trained model vs ONNX Runtime: max |diff| 8.94e-07, argmax agreement 1.0000 [export] OpenCV DNN (opencv-dnn(new engine)) vs ONNX Runtime: max |diff| 0.00e+00 -> OK

A one-folder deployable bundle with a parity check and checksums

What we learned

Better inputs beat bigger networks for look-alike letters.
Evaluate on a signer the model has never seen, or the numbers lie.
"The model proposes, a validator disposes" works well for small generative models.
Time-based decoding keeps behavior stable across cameras and frame rates.