Verbatim

Inspiration

One of our teammates is friends with someone who is deaf and has seen firsthand the challenges of communication, especially due to the lack of universality of ASL. This inspired us to create a tool that makes ASL more accessible and to bridge language gaps while also turning the process into a cool visualization!

What it does

Verbatim translates ASL in real time and visualizes it in a fun, creative way through audio-reactive particle motion. As users sign, their words are spoken aloud while simultaneously generating dynamic visuals, creating an engaging and immersive communication experience.

How we built it

We used Google’s Mediapipe and OpenCV to detect and track hand landmarks, which we then processed into a dataset and trained using a Random Forest model from scikit-learn. Using this trained model, we built a live ASL-to-text translator paired with text-to-speech (pyttsx3) to vocalize the translated sentences.

To enhance the experience, we also implemented an OSC bridge (python-osc) to connect our system to TouchDesigner, where we created an audio-reactive simulation that responds in real time as the translation occurs.

Challenges we ran into

Many of our challenges stemmed from the configuration of programs as well as the implementation of new skills. For these reasons our project went through a lot of changes within the short time frame of the hackathon. For example, we transferred from using Tensorflow, Keras, and the Kaggle dataset to using Scikit Random Forest instead due to our small datasets and necessity to quickly train our own data.

Another challenge occurred when our computer overloaded and deleted all the work we had done on our TouchDesign interface (5 hours of work). However, our extensive documentation, notes, and planning allowed us to restore everything in approximately 1 hour.

Accomplishments that we're proud of

One accomplishment we are incredibly proud of is the connection between our ASL detection and TouchDesigner. In our past as individual developers we often ran into issues where two programs work separate from one another but we fail to cohesively run them together. Today however, we managed to do just that.

We are also immensely proud of how we handled our entire system deleting itself. Our system of planning, with color coded segments not only allowed us to properly divide tasks as we worked on them independently, but it also allowed for the perfect way to “retrace our steps” when we hit technical roadblocks with TouchDesigner deleting all its data.

What we learned

We learned how to use an entirely new platform, TouchDesigner, and were able to successfully implement it into a working system despite having no prior experience.

Additionally, we learned how to adapt quickly, manage stress under pressure, and the importance of thorough documentation. These skills allowed us to recover from setbacks and continue building effectively.

What's next for Verbatim

We plan to expand our word database beyond fingerspelling to support more complete and natural ASL communication. We also aim to create a more cohesive program by combining all the elemental UI together into one streamline platform.

We also aim to collaborate with the deaf community to ensure our tool is accurate, meaningful, and truly helpful. In the future, we hope to integrate Verbatim with platforms like Zoom to enable real-time translation in virtual environments and improve accessibility in online communication.