Inspiration
We Sea You is an inclusive application aimed at visually impaired users. It intends to give the visually impaired individuals experience of the underwater world by translating their surroundings to speech. This will allow them to carry out recreational activities like scuba diving and also give them access to more jobs (such as that of a marine researcher) in the scientific community. I was inspired to create this app on reading an article ‘To Sea With a Blind Scientist’ from Geerat J. Vermeij, a nationally recognised marine biologist. In his article, he questions ‘could a blind person ever hope to be a scientist?’ On reading the article, I realised that there is a lack of opportunities in the scientific research community for visually impaired individuals, and this motivated me to bridge this gap by creating We Sea You.
What it does
The app detects the surroundings and translates it into speech making it possible to explore the underwater environment. I have made sure the app is realistic by considering the following points -
- The ML model will be available offline on the app for object detection.
- There exists mobile technology as well as speakers that are waterproof and can be fitted into a full face mask. Hardware considerations reference
- As an alternative, visually impaired users can use submersibles to explore the sea.
How I built it
We Sea You app is built using Flutter for an android device, although, I plan to extend its functionalities to an iOS platform as well. The app uses a TensorFlowLite API to detect objects in the scene (with the help of tflite plugin). The labels are used to generate text which is converted to speech using a Flutter plugin for Text to Speech (flutter_tts). The flutter_tts plugin also supports multiple languages, and therefore the use of the app is not restricted to English-speaking users. The ML model used for object recognition is Mobilenet_V2_1.0_224_quant and the model is hosted on Firebase.
Challenges I ran into
Translating surroundings from a real-time video to speech, frame by frame has been a challenge, especially since I didn't have much experience with Flutter before creating this project. Furthermore, managing deadlocks between concurrent processes such as image streaming, ML object detection and speech generation was quite demanding and is something I am still working on. Although, this has given me a better understanding of what deadlocks are and how to manage them. Finally, I would love to test my app underwater and I am upset I haven't been able to because of the COVID-19 situation :(
Accomplishments that I'm proud of
Even though I have not been able to complete the demo video for the app, I am proud of what I have learned and accomplished in the past 48 hours, but I do hope to get better at managing my time in the upcoming hackathons. (I will be posting a link to the video soon :)
What I learned
Stick to coding in python :)
What's next for We Sea You
I plan on increasing the functionalities of the app by including a speech to text API to allow the users to record their experiences. I also plan on training my own ML model with images of different marine creatures. Another possible extension can be including a dataset of description for each of the detected objects in order to give the user more information about the characteristics and nature of the object.
Log in or sign up for Devpost to join the conversation.