Inspiration
There is a huge gap between people who are deaf/mute or both and people who cannot use sign language to communicate with them. We wanted to bridge this gap between the two populations with this app.
What it does
It is a text-to-speech and speech-to-text application that lets the user (Deaf/Mute/Both) communicate with someone who doesn't know how to interpret or use sign language. The user can type out a query on his phone, and the text thus typed is said out loud to the other person, who can listen to it and reply to that using spoken language. What the other person says is then converted to text using a speech-to-text API, and displayed on a dedicated section on the user's screen.
How we built it
The tech stack we used was Flask on Python and Flutter with dart on the front-end, our back end has an API to handle all the Text-To-Speech, and an API to handle basic user authentication, user logins have not been implemented but anonymous users are used for a new device which uses our application.
We primarily used the flutter packages, flutter_tts and speech_to_text to handle all the text to speech and speech to text, the package uses localised speech to text/ text to speech conversion so it has extremely low latency, and also has much higher accuracy rate than a manually coded model as the trained model provided by the mobile providers is extremely well trained due to the extensive data sets they harbour.
The UI was probably the most tedious part of the process, and we used the Apple Design (Cupertino) Principles to build it, in flutter.
As Flutter is a reliable frontend framework, as it is maintained by google, and also the fact that it gives you multi-platform applications, makes it a very versatile application
Challenges we ran into
the text to speech and speech-to-text both had a buffer before they started working, making it really hard for the user to maintain fluid conversations.
Widget placement was also an important part of it, and took numerous hit and trial attempts to get to a feasible position for us to place them
Accomplishments that we're proud of
India's first such app in existence, to some extent, even the world's first
What we learned
figma design, apple design principles, working with tts and stt, much more.
What's next for Indri.yeah
scaling it to have chat history and even an AI based application to detect and understand sign language.
Log in or sign up for Devpost to join the conversation.