Sign_Interpreter_App

Sign language interpreter live running demo

Inspiration

Computer vision scope. We've always had this vision to develop a software which uses human interactions to automatically perform computer actions without the use of non-renewable hardware. In essence, super efficient communication with our non-living significant others. Our inspiration also comes from the latest advancements in spatial computing to efficaciously fuse the physical with the virtual world.

What it does

Converts real time hand gestures to letter based on ASL. Our highly interactive animated environment inputs real time hand signs (sign language mostly) and converts that into text in a brisk pace. We are able to generate letters to words to sentences to even paragraphs with this amazing tool.

How we built it

Model Accumulation - We acquired a brilliant python model to effectively map different signs on hand through coordinate development. Inspired by the YOLOv8. model.
Model Training - Our favorite part was actually training the model to input the different hand gestures for each corresponding letter. We had to click almost a hundred pictures in various angles for each letter! Very important that we train our model right!
UI Dev - Our real time consuming part... Using react.js, we needed to upload various pictures and gifs and make sure we got each position, size, font, etc. absolutely spot on
Combining code and UI - We combined our backend python model with our front end react interface. Moreover, we animated most of our website by incorporating the theme of the hackathon - RowdyHacks - by designing moving elements

Challenges we ran into

Having real time conversion - It was a tedious task taking real time hand inputs and output a unique letter with almost no latency
Effectively combining the backend code with the frontend part and trying to input and project real time string characters to our end user.
The weirdest difficulty of them all - our group members all had laptop screens with different aspect ratios so elements on our webpage were misplaced and oddly shaped majority of the time for the same lines of code

Accomplishments that we're proud of

We are very proud of our time management and the kind of web program we were able to develop in such a short period of time. Some of our best accomplishments include:

Processing live automated typing from camera inputs with a significantly low latency.
We acquired in depth knowledge of various programming languages and tools such as JavaScript, Python, React, ML Algorithms, etc.
Time Management and delegating tasks amongst our team of skilled coders.

What we learned

We almost only learned throughout our journey for this hack. As we went with the development for our code, we learnt various new commands and sequences for efficient programming and certain hacks to manage our time better with specific libraries and AI tools. I believe our biggest learning outcome was debugging - The most strenuous and time consuming job in coding. We organized our codes in properly named files with a convention and organized our hundreds of lines of code with proper print statements and comments for ease of debugging.

What's next for the "Sign_Interpreter_App"

The most important aspect we need to work on is the Webpage/User Interface. Our future goals are to create a more interactive webpage and try to launch it for users who prefer to perform signs directly with their hands instead of typing on the keyboard. Another thing we would like to work on for this interpreter is adding numericals and other characters instead of just letters to make our program more diverse and deep. Last but not the least we will ensure that we can add more human-computer interactions broadening our horizons from just sign language itself!