Mathla — Multilingual Sign Language Intelligence

Mathla — About the project

Where this started

Mathla didn't start in a lab. It started at home.

Someone in my family is deaf. I grew up watching a brilliant, funny, complete person be treated as difficult to talk to — watching rooms slow down for them, or worse, move on without them. Not because they lack language. They have a rich, full language. The world just never bothered to learn it, and none of our technology did either.

A hearing person can mumble at a phone in any accent and be understood. A Deaf person, in 2026, still cannot stand in front of a hospital counter and simply say what hurts. That gap is not a technical inevitability. It's a choice the industry made about whose communication matters. Mathla exists to unmake that choice — starting with Arabic, the language of a region where this gap is widest and the data is thinnest.

What Mathla is

Mathla is a continuous sign-language translation system: a person signs naturally at a phone camera, and the system produces written and spoken language.

Not ten fixed gestures. Not a demo where you pick a phrase from a menu. The goal — stated plainly — is open-domain, continuous sign-to-speech: letters, numbers, fingerspelled names, single words, full sentences, and natural combinations of all of them, signed the way Deaf people actually sign, understood without the signer changing anything about how they communicate.

We build one brain per language — Arabic first, English as its own separate system — because languages deserve their own models, not a blender. And we are building the reverse direction too: speech and text rendered back as sign, so the bridge carries traffic both ways.

What we have actually built — honestly

One year in, this is real and running:

A working bounded translator. On our fixed evaluation of known Arabic sentences, the production system scores 199/200 with zero wrong-spoken outputs, verified by an automated regression gate that runs before anything ships. A deaf person signing supported sentences at a phone hears correct Arabic speech, live. A full data ecosystem. A public platform where the signing community records letters, numbers, words, and sentences; reviewer workflows; consent management; and a capture format that stores the live landmark stream and the raw video of every recording — so today's data stays usable by tomorrow's better tools, forever. A unified capture engine. The exact same code path that powers live translation also powers data collection, on web and mobile. What the brain eats in training is what it receives in production. We learned to enforce this the hard way. A research lab that doesn't lie to itself. Roughly 150 logged experiments with pre-registered predictions, signer-disjoint and sentence-disjoint evaluation only, and a standing rule that any comparison must state exactly what each system received as input. We've caught our own measurement bugs — truncated inputs, asymmetric probes, corrupted tensors — before they became false claims. Several of our proudest-looking numbers died under honest testing. We filed the deaths.

What doesn't work yet — same honesty

Open-domain is not solved. On sentences the system has never seen, our honest error rate is still far too high to ship — and after exhausting every architecture trick, recipe, and data remix we could invent, we proved why: the model isn't missing a technique. It's missing unique signed sentences. No decoder on earth can produce sentences its training data never contained.

So the frontier of this project is not a bigger model. It's a recording campaign: thousands of new Arabic sentences, each performed by multiple verified Deaf signers, captured through the same pipeline the product runs on. Reference videos — recorded by Deaf teachers, so contributors imitate a trusted demonstration instead of reading text — make the campaign accessible to the many members of the community who were denied literacy by the systems that failed them. The community isn't the subject of this dataset. They are its authors.

What we learned that we didn't expect

The measurement lies before the model does. Every major false result in our history traced back to the ruler, not the brain — a truncation constant, a rigged comparison, a corrupted input stream. We now audit the ruler first, always. Fingerspelling is its own frontier. Words and numbers transfer across signers; letter handshapes barely do. The information lives in the fingers — the exact place cameras and landmark extractors are weakest. We've opened a dedicated front on it, and hand-centric features have already moved it double digits. Volume beats grinding. More real data helped every time. More epochs, more augmentation, more clever optimization on the same data — almost never. We wrote it down as a law and it now vetoes experiments before they waste a GPU-hour. Trust is architecture. A translation system for the Deaf must never confidently speak the wrong thing. Our system is built to be silent when unsure — because a translator that is right 15 times and silent 5 is usable, and one that is right 18 and wrong twice is dangerous.

What Build Week added

For OpenAI Build Week, we used Codex and GPT-5.6 to make a year of intense, sprawling work inspectable: reconstructing experiment history from logs, mapping system components, and building the Mathla Research and Evidence Agent — a GPT-5.6-powered agent that answers questions about the project ("what's verified? what failed and why? what's operational vs experimental?") from a sanitized evidence base rather than model memory, reporting verification status with every answer. The same honesty discipline we enforce on our models, now enforced on our own claims.

The goal

A Deaf woman signs a question at a government counter and is answered like anyone else. A student raises his hands in class and the teacher understands. A father tells his hearing daughter a bedtime story in the language of his hands, and she hears it.

We are not there. We are honestly, measurably, on the road — with a working bounded product, a community pipeline, a research engine that tells the truth, and a region full of people who have waited long enough.

A person signs naturally. Mathla understands. The world receives the message.

Built With

• android
• api
• codex
• continuous
• ctc
• gpt-5.6
• javascript
• language
• laravel
• learning
• natural
• openai
• php
• postgresql
• processing
• python
• pytorch
• react
• recognition
• redis
• rest
• sign
• typescript
• vision