Introduction
Hand Speak👋– Sign Language Recognition is a system designed to bridge the communication gap between the hearing and deaf-mute communities using computer vision and deep learning. The system captures hand gestures through a camera, processes them with machine learning algorithms, and converts them into real-time text output. This Project supports two Sign Languages which are A). American Sign Language (ASL) and, B). Indian Sign Language (ISL).
Objectives
- To develop a sign language recognition system using computer vision and deep learning.
- To achieve real-time processing with minimal delay.
- To create an accurate and scalable model that can recognize a wide range of sign language gestures.
- To build a user-friendly web application for ease of access.
Proposed Methodology
- Capturing Hand Gestures: A webcam or external camera is used to capture hand movements in real-time.
- Preprocessing the Image Frames: The captured frames undergo preprocessing, such as resizing, noise reduction, and color correction, to enhance recognition accuracy.
- Feature Extraction: Key points of hand gestures are detected using computer vision techniques like MediaPipe or OpenCV, and features are extracted for classification.
- Gesture Classification: Deep learning models, such as Convolutional Neural Networks (CNNs) are used to classify gestures.
- Text Output Generation: Once a gesture is classified, the system maps it to the corresponding word or phrase and displays the output text.
Challenge Analysis
- Overfitting in model: Small dataset prone to overfit AI model. Solution is increase the data size.
- Misclassification: Some time model does not get correct output for hand sign. So we set the threshold value with 10 correct prediction.
- Similar Sign: Different alphabet (symbol) have very similar Sign. Set A: {M,N,S}, Set B: {D,R,U}.
- Noisy Feature: Hand Size, Hand Colour are noisy feature for AI model. Solution is include the Media pipe Landmark system in project.
Accomplishments that we're proud of
- Successfully developed a real-time Sign Language Recognition system using computer vision and deep learning.
- Enabled seamless communication between Deaf/Mute individuals and hearing persons.
- Promoted inclusive technology by creating a tool that supports accessibility and social impact.
What we learned
- Learned and implemented key technologies such as OpenCV, TensorFlow, Keras and MediaPipe.
- Trained and tested the model with high accuracy using a well-structured dataset of hand signs.
- Gained real-world experience in AI for social good.
What's next for Hand-Speak
- Real-Time Translation: Improve the system to support continuous gesture-to-text translation in real-time.
- Voice Output Integration: Add speech synthesis so translated text can also be spoken aloud.
- Multilingual Support: Incorporate translation of sign language to different spoken/written languages.