Inspiration

I remember being at a public event at my school library where my friend was able to communicate with a girl who only spoke sign language but I wasn't. I felt a bit left out, however, I saw that girl struggling to communicate teachers and speakers at the event and I wanted a way to mitigate that barrier. Sign language is different from other languages with the fact that many applications like Google haven't developed translating systems with it. Google translate covers a lot of spoken languages but who covers Sign Languages? Thus translating sign language just like a person would've been able to translate a spoken language using applications like google translate sounded like a great idea to me. That's where my inspiration for SignStudio ultimately came from.

What it does

It uses a user's camera to analyze hand gestures like signs and translates them using pre-defined words that the user has already created blueprints from. It not only is able to translate ASL words but is able to format them in a sentence-like manner that can be effectively saved and retrieved later. Additionally, there is a creative studio that is able to allow users to create their own gestures or signs that they use daily. Examples of this can be secret languages shared with siblings or hand gestures that are taught in hospitals to employees or doctors to signal certain situations. It essentially is a way users can create, translate, and form sentences with hand gestures.

How I built it

I used Streamlit to build it using mostly Python and I also used CSS to design the UI and a config.toml file as well. I used specific libraries like mediapipe for the joint connections, numpy for the recognition standings (recognition engine), and openCV for the camera that is able to collect the signs. I also used JSON files to store the coordinates for the handprints that were collected through the blueprints the user inputted and put those JSON files inside folders dedicated for each specific category. The Code editor I used was VSCode and I used Github as well as Streamlit to deploy it.

Challenges I ran into

Deployment was a big pain for me. because I used OpenCV, streamlit wasn't able to support it. I consumed around 5-6 hours trying to fix that but it didn't get fixed in the end. I couldn't deploy it with a working link which I aim to mitigate later. Additionally, mediapipe versions and usages were difficult because it was difficult to make the joints and also create a json file for each blueprint.

Accomplishments that I'm proud of

I'm glad that I was able to accomplish what I wanted in the end, even if it's local, I liked that the translations were accurate and the recognition engines were able to show the standings well, proud to have shared this creation with United Hacks v7

What I learned

I learned more about streamlit and deployment- do not use openCV unless you want to run into issues. It was too late when I realized but instead of using openCV, streamlit has it's own webcam, which although it is less in quality, can also be used for easier deployment. I also learned how to use JSON files.

What's next for SignStudio United Hacks v7 Project

Deployment for sure. I want to deploy it and be able to use it in ways. I might present at my local library and try to create impact using Social Media like Instagram with a team that is able to not only improve the product but is able to gain support for it through advertisement and use of the app.