Sign is more than an educational app, it’s a communication tool. Our development team was inspired by language apps that incorporate gamification like Duolingo, and educational language apps like Rosetta Stone. With a minimal user-interface, combined with a camera and video scan technology, Sign is great for those looking to learn sign language, and those looking to communicate using signing in everyday conversation.

What it does

Incorporating voice-to-text technology, and text-to-image processing, Sign translates spoken word to text and text to sign language, creating a fast and conversational way for beginners to learn sign language and an interactive way for more experienced signers to brush up on their skills. Sign also offers three tiers of language complexity so you can advance in your learning.

How we built it

We used Miro and dot voting for ideation and convergent thinking. Low, medium, and high fidelity prototypes were built in Figma. The app was built using Javascript, React Native, Android Emulator, and Expo.

Challenges we ran into

We weren’t able to implement all the functionality we envisioned such as text to sign, and camera scanning. Our development team took on the challenge of working with new tools and languages such as javascript, react native, android emulator, and expo. Other challenges include managing version control history, adding audio recorders and listeners, adding camera functionality, incorporating IBM API, and integrating the speech to text translation to our react native app using IBM API, The team had debugging issues that took up a substantial amount of our development time. Another struggle was working remotely from home and the technology limitations this presented.

Accomplishments that we're proud of

Our development team had limited prior knowledge of mobile development prior to the hackathon. Given this, we’re proud of what we were able to accomplish in a short period of time, as well as the teamwork we exhibited throughout the process. We were able to develop a high fidelity prototype, and create app functionality like adding audio and listeners, adding camera functionality, translating voice to text using IBM API, creating mobile navigation, a splash screen, building a login system page, learning how to use gradient color on apps, and sending debugging request to service. We’re also proud of our idea which we believe if implemented, has the potential to make a positive impact.

What we learned

Our development team learned new technologies by using tools like react native, android emulator, expo, and IBM API. In particular, how Expo has made development easier and more intuitive (e.g. publishing app, app technologies such as audio and camera, multi-platform compatibility is automatic, React makes the app-building a little more intuitive). During Gear up we learned about different company APIs, and we learned how to incorporate IBM’s API into our project. We also made use of virtual project management tools and ideation tools like Miro and Lettucemeet. Our programmers learned about Figma and Design thinking and our designer learned about coding and programming, we feel this is valuable for better combining these two skill sets in future development work. We also learned a little bit of sign language!

What's next for Sign

What’s next for Sign? We weren’t able to fully develop all the functionality we envisioned. If Sign were to grow, we envision including increased levels of complexity beyond short sentences. We would also incorporate 3D technology and animation to better show hand movement, which is crucial in sign language. We would further develop gamification in the app to make the experience more engaging for learners. Future iterations would include user research upfront as well as usability testing. Our prototype also shows a scanning feature with the purpose of confirming a user’s comprehension. We would continue working on our UI to make the app more user intuitive. We would also implement the functionality we weren’t able to implement.

Prototype demo:

Share this project: