• Due to its relative youth and widely dispersed user base, American Sign Language (ASL) has never developed standardized vocabulary for the many terms that have arisen in advanced Science, Technology, Engineering, and Mathematics (STEM) fields. This makes it hard for deaf students to learn in their native language, and it makes communication between both deaf and hard of hearing scientists and engineers far more difficult.
  • As any intro linguistics student could tell you, language use and evolution cannot be dictated by the few, no matter their expertise. This approach has failed time and again throughout history, and, no doubt, this case will be no exception. Instead, languages change because their users choose to change them.
  • This project is an attempt to connect ASL users together, so that we can introduce the necessary vocabulary to the language. This will make it much easier for those in the Deaf community to pursue careers in technical fields.

Combating Covid-19

  • Online education with a reliance on un-captioned videos or audio hurts the deaf community
  • Zoom calls and webcasts are very difficult to effectively sign over
  • This hurts the ability for some students to learn in their native language
  • Our app enables classes to share and develop vocabulary *Enabling people to communicate and develop signs about new terms, such as “COVID-19”
  • Empowers communication about the newest terms whose signs haven’t developed yet
  • A global, online resource for language development
  • Many existing audio-reliant resources are not designed or accessible to the deaf community making the transition all the more difficult
  • Our app strives to provide more online and accessible resources to the ASL community in this time of need

What it does

  • StemSigns serves as a visual dictionary of STEM terms for the deaf community
  • We provide an existing database of short videos demonstrating STEM terms via ASL
  • The community is also encouraged to submit their own videos providing possible ASL translations for words
  • These new videos will join the StemSigns database and help broaden / expand ASL
  • We also provide an option to receive daily text updates to encourage the learning of a new word a day
  • StemSigns also keeps a hierarchical list of categories and related words, allowing for easy navigation and expanding of one’s own vocabulary

How we built it

  • The app is built in react.js as a website (with the requisite HTML/CSS)
  • We used GCP for user authentication/sign-in and Firebase for the word/definition database
  • Twilio enables us to send Words of the Day to users, encouraging people to keep learning new things!
  • We have a MongoDB Atlas backend for our user/phone number database
  • Youtube’s Data API (GCP) was used to allow for instant user uploads to a community account
  • HTML5 Webcam support
  • UIPath for error detection and early warning
  • Material.ui for the UI/styling

Challenges we ran into

  • Took a lot of time to figure out OAuth authentication for Youtube uploads
  • The dataflow from Webcam->Local->Youtube->Backend was decently complex/tricky
  • Styling was a bit of a challenge since we had no one with that expertise on our team
  • The Database/Backend took a bit of time as we were having trouble having the twilio side working

Accomplishments that we’re proud of

  • Instant, easy to use youtube uploads!
  • Live, user-updatable website
  • Direct Webcam->Website dataflow
  • Fully functional search
  • Automated Word of the Day support via Twilio
  • Clean Material.ui
  • Accessible design

What we learned

  • Accessible design goals/principles
  • More experience with react js
  • Youtube Data API
  • Material.ui
  • HTML5 Webcam/Recording workflow
  • MongoDB Atlas
  • FireDB/Databases in general
  • UIPath for warnings
  • Twillio!
  • OAuth2
  • Formatting Post/Get Requests

What's next for StemSigns

  • Language support beyond English
  • Pictogram/pictoral support
  • Adding comment support
  • Voting on the best video definitions
  • NLP to find related words
  • Better Admin tools for moderation and removal

Built With

Share this project: