HumKalaam

• 

  HumKalaam

• 

  Text/Audio to Sign

• 

  Avatar Functions

• 

  Sign to Audio/Text

• 

  SIgn to Audio Details

Inspiration

Pakistan has very few sign language interpreters, so most conversations between Deaf and hearing people happen through written notes, or not at all. We wanted a two-way tool on the phone people already carry. "Hum Kalaam" means "talking together".

What it does

• Text or voice to sign: type or speak English, and a 3D avatar signs it in ASL. 476 signs, including everyday Urdu words like chai and ghar. Unknown words are fingerspelled, never swapped for a similar sign.
• Sign to text: sign to the camera, and the app writes it out and reads it aloud. It knows 20 everyday words out of the box, and you can teach it more.
• Everything runs in the browser, and the video never leaves your phone.

How we built it

We built on the open-source three.ws signing engine and added 444 everyday signs, a text and voice pipeline, and a phone-style interface in plain JavaScript, three.js and Vite. MediaPipe tracks the hands in the browser, and our recogniser matches movements against each sign. We built it with AI: the Dev Marketplace IDE (Kimi and Haiku) to prototype and set up, and AI coding assistants to write and debug, while we made the decisions and tested every version.

Challenges we ran into

• At first most words were fingerspelled, because the engine had only about 30 signs. We expanded the vocabulary to 476.
• The camera took over a minute to load from slow servers. We bundled the models, and now it loads in about 4 seconds.
• Recognising 476 signs from a webcam was unreliable, so we focused on 20 clearly different words (99% in simulation).
• A 47 MB avatar made phones lag, so we switched to a lighter one.

Accomplishments that we're proud of

A truly two-way app that never shows a wrong sign, never guesses, and works on an ordinary phone with nothing to install.

What we learned

Sign language lives in handshape, location and movement, not just words. Doing 20 words well beats doing 476 badly. And AI speeds up building, but a person still has to test everything.

What's next for HumKalaam

Review by Deaf signers, Pakistan Sign Language, a larger camera vocabulary trained on real signers, and a pilot with a Deaf school and a clinic.

Built With

• asl
• computer-vision
• css3
• dev-marketplace
• dynamic-time-warping
• getusermedia
• gltf
• hand-tracking
• html5
• indexeddb
• inverse-kinematics
• javascript
• mediapipe
• node.js
• pose-estimation
• sign-language
• speech-recognition
• text-to-speech
• three.js
• three.ws
• vite
• web-speech-api
• webassembly
• webgl