Inspiration

The inspiration for building a sign language detection system came from the desire to improve communication and accessibility for individuals who are deaf or hard of hearing. Sign language is a visual and gestural language, and a sign language detection system could help bridge the communication gap between individuals who use sign language and those who do not. Additionally, such a system could be useful in educational settings, healthcare, and other areas where communication with individuals who use sign language is important. The advancement in computer vision and deep learning techniques, allows for the development of more accurate and efficient sign language detection systems.

What it does

It is a web application that can recognize and interpret sign language gestures. It typically uses a combination of video input and machine learning algorithms to detect and identify specific signs being made by a person. The output of the system is in the form of text. Depending on the specific application, the system could be used in real-time, such as in a conversation between two people, or it could be used to analyze images. The system could also be integrated into other technologies, such as virtual assistants or smart home devices, to make them more accessible to individuals who use sign language.

How we built it

Building a sign language detection system typically involves several steps:

Data collection: The first step is to collect a large dataset of sign language gestures. This was done by using pre-existing datasets. The dataset includes a variety of signs, different signing styles, different lighting conditions, and different backgrounds.

Preprocessing: The next step is to preprocess the data. This includes cropping the video to just show the hands, normalizing the lighting and color, and resizing the images to a consistent size.

Model training: Once the features have been extracted, the system can be trained using a machine learning algorithm. This typically involves using a supervised learning approach, such as a neural network, to learn the relationship between the features and the corresponding sign.

Testing: Finally, the system should be tested on new unseen data to evaluate its performance and make any necessary adjustments.

It's worth noting that building a sign language detection system is a complex task, and that the specific methods and techniques used may vary depending on the application, the data available and the level of accuracy and efficiency required.

Challenges we ran into

There are several challenges we encountered when building the sign language detection system:

Variability in signing styles: Sign language can vary greatly between different individuals, regions, and cultures, which can make it difficult for a system to recognize and interpret signs accurately.

Occlusion: Hands and fingers can often be occluded by other body parts or objects, which can make it difficult for the system to detect and interpret the signs being made.

Lighting and background: Sign language gestures can be affected by lighting conditions and the background, which can make it difficult for the system to detect and interpret signs.

Real-time performance: When building a real-time sign language detection system, it can be challenging to achieve a high level of accuracy while also maintaining a fast processing time.

Limited data availability: Sign Language is not as widely spoken as other languages, and so there may be limited availability of data and resources for building the sign language detection system, which can make the process of developing and training models more difficult.

Overall, building a sign language detection system can be a complex and challenging task that requires a combination of computer vision and machine learning expertise, as well as a good understanding of the nuances of sign language.

SignSpeak (Indiana Codes - Team 156)

A real-time sign language translation tool

Motivation
Hand Detection
Detection Output
Tech Stack
Functionalities
To Do and Further Improvements
Requriements
Run Locally
License
Contributors

Motivation

Physically impaired people find it hard to communicate with other humans or latest computing innovations in conversational AI. They normally communicate via sign languages which is often a mix of hand gestures and facial expressions. A computer interface which can interpret their hand gestures from a single camera view and convert it into text or speech transcript would be really beneficial for both parties. We are dealing with static signs for simplicity but a similar concept can be utilized for developing dynamic sign recognition. This system can help in effective human computer interaction for impaired persons which can be utilized for Customer Support, virtual meetings etc.

Hand Detection

The crux of this solution lies in identifying the presence of a hand in the video feed/image. This is done using various modules like OpenCV, cvzone and mediapipe.

Home Page

FAQs

Detection of a character/digit in an image

Character/digit recognised

Detection Output 1

Detection Output 2

Detection Output 3

A single frame from a video

animated

Tech Stack

HTML, CSS and Bootstrap have been used for the front-end
Flask has been used as a back-end framework
The application has been developed using Python

Functionalities

Detect a hand shown in the video feed/image.
Draw a box around the hand which has been detected.
Display the character/digit recognised.

To Do and Further Improvements

[x] Using OpenCV, cvzone and mediapipe for hand detection
[x] Develop an algorithm which can automatically classify sign language from a video taken from a webcam and convert it to text/speech transcript.
[x] Display the character/digit shown
[x] Detect and draw a box around a hand (if present) for an image, video/live stream.
[ ] Adding a button to turn on/off video feed
[ ] Updating the simple and minimalistic UI

Requirements

The following dependencies and modules(python) are required, to run this locally

opencv-python==4.6.0.66
flask==2.2.2
wtforms==3.0.0
flask_wtf==1.0.1
tensorflow==2.9.2
mediapipe
cvzone

Run Locally

Clone the GitHub repository

$ git@github.com:DeathStroke19891/signspeak.git

Move to the Project Directory
```
$ cd sign-speak
```
Create a Virtual Environment (Optional)
- Install Virtualenv using pip (If it is not installed) python $ pip install virtualenv
- Create the Virtual Environment python $ virtualenv sgnspk
- Activate the Virtual Environment
  - In MAC OS/Linux python $ source sgnspk/bin/activate
  - In Windows python $ sgnspk\Scripts\activate

Install the requirements

(sgnspk) $ pip install -r requirements.txt

Run the python script main.py
```
(sgnspk) $ python3 main.py
```
Dectivate the Virtual Environment (after you are done)
```
(sgnspk) $ deactivate
```