Inspiration
It all started with a simple question: Why do people who are blind or visually impaired have to struggle with traditional typing on a screen, where they aren't even sure if they pressed the right keys? Traditional text input requires physical keyboards or tedious screen-reader navigation, which can feel uncertain and disconnected. I realized that a more natural, human-centered approach would be to let users simply sign language in front of a camera, having their gestures automatically converted into real-time text. To ensure complete confidence and accessibility, I integrated audio feedback, allowing users to hear the exact alphabet or letter they just signed before it gets typed out. By focusing specifically on Arabic Sign Language (ArSL), this project bridges a major regional gap in assistive technology.
What it does
The ArSL Auto-Typer is an end-to-end computer vision and machine learning application that runs entirely through a standard webcam. Real-Time Landmark Extraction: It uses Google MediaPipe to track dual-hand movements simultaneously. Intelligent Auto-Typing: Once a valid sign is detected with high confidence, intelligent cooldown timers prevent duplicate spam, and the system automatically types and pastes the Arabic text straight into any active window (like Notepad). Audio Validation: It incorporates automated text-to-speech feedback so users have instant audio confirmation of what letter was recognized.
How we built it
Dataset Collection & Training: Built custom Python scripts using OpenCV and MediaPipe to capture 3D hand landmarks for Arabic alphabet signs, training a robust machine learning classification model (model.p and classes.npy).
- Computer Vision Pipeline: Processed live video frames, flipped them for a natural mirror view, and extracted 126 coordinate features across dual hands.
- System Automation: Used scikit-learn for probability prediction, combined with pyperclip, pyautogui, and custom threading to handle background text injection and audio synthesis seamlessly.
Challenges we ran into
Dual-Hand Tracking Precision: Ensuring the model could dynamically handle single-hand versus dual-hand signs without crashing or misinterpreting feature lengths. Typing Spam Control: Implementing strict confidence thresholds and cooldown delays so that holding a sign wouldn't flood the active text document with endless duplicate letters.
Accomplishments that we're proud of
Successfully building a fully functional, hands-free Arabic Sign Language translation and auto-typing pipeline purely through software. Creating an inclusive tool that empowers users with a smoother, audio-backed alternative to traditional screen typing.
What we learned
We gained deep experience in real-time computer vision, managing multi-threaded audio processing, and fine-tuning machine learning classification models for real-world assistive applications.
What's next for ArSL Auto-Typer & Voice Synthesizer
Built With
- mediapipe
- opencv
- python?
