Totomi (Talk To Me)

Totomi

Inspiration

The idea of Totomi originated from an urgent email. A student was looking for a notetaker for a class because he was suffering from hearing disability.

About 37.5 million American adults report troubles with hearing. That is about 15% of the American population. Furthermore, more than 360 million people suffer from disabling hearing loss across the globe. [source: US Department of Health & Human Service, World Health Organization].

Therefore, we wanted to develop a tool, a seamless integration with their daily life, that can help facilitate everyday conversation with the world. So that our classmate could take a class without needing a notetaker.

What it does

Totomi is a mobile application that records and transcribes the conversation into text. There are few services that provides similar functionalities, but most of them focuses on phone conversation, needs an operator to listen and type out the conversation, and/or requires a separate hardware.

Unlike these complicated, high cost solutions, our team utilized newly emerging machine learning libraries that automatically transcribes an audio stream/file within a single mobile application. Now, people can easily access our service without needing a separate hardware or having to worry about service availability.

How we built it

We used React-Native library to support both Android and iOS devices. React-Native-Voice library invokes machine learning library that processes the audio file in Google Speech API or Apple Speech Recognition API. We chose to build mobile application, because more than 77% adults in United States owns smartphones and more than 4.78 billion people in the world has access to smartphones. Because it was developed through React-Native library, Totomi has access to both Android and iOS communities, which increases accessibility of the app. We wanted the design to be clean, functional, extremely organized, and easy to use. The design and implementation of our application reflects that goal to reach out to more people with hearing disabilities.

Challenges we ran into

We decided to build our app with React-Native library, but we found out that React-Native is not fully developed library compared to iOS SDK or Android SDK. We found out that some of key functionalities are not provided us as a built-in functions. Luckily for us, a large community of developers and hackers participated in open source projects that gave us few selections to gain access to few key functionalities. But the process of configuring each library and choosing right type of library was very challenging. Also we wanted to implement Google Cloud Platform on the mobile application. But Google Cloud Platform does not provide an SDK for React-Native. Although it supports Node.js SDK, we found it very hard to integrate it with React-Native application and not much documentation on it.

Accomplishments that we're proud of

One of our the most accomplishment is that we had a creative idea to reach out to people who may have been marginalized from tech communities. Our market research showed that, although there existed many hearing disability applications, hardware and support services, there was not any products that utilizes machine learning and cloud computing. We wanted to utilize those computer science miracles to help more people. We are also proud to complete the prototype with fully functional speech recognition and beautiful futuristic UI.

What we learned

We learned that React-Native is very useful tool that could develop both iOS and Android app; however, it comes with a cost of incredibly difficult environment, missing basic functionalities and lack of official support tools from third parties. But we will continue develop our app on React-Native.

What's next for Totomi

The potential of the app Totomi is endless. Totomi was never intended to stop after GTHack. With continuous support, our team is going to add more features to Totomi such as Text-to-Speech, Sign Language to Text / Speech, Speech / Text to Sign Language by utilizing more machine learning tools and cloud resources.