Dyslexia is a language-based learning disability or disorder that includes poor word reading, word decoding, oral reading fluency and spelling. It is estimated that 1 in 10 people have dyslexia. We have witnessed many dyslexic people having to be dependent on others to a great extent when it comes to education. We aim to do something about this issue and make these people more independent. We came up with an idea to make Self-Learning possible for them.
What it does
We have created a website which takes in input of three formats, namely, Text, Speech and OCR images and gives output in the form of music and visual images. This is helpful to a great extent because people with dyslexia are a lot better with interpreting audio-visual data and music. It helps them create a link in their brain and remember that particular information for a longer period of time.
How we built it
We first convert all out input format to text format. The Google-Cloud-Speech-to-Text API is being used to convert the speech into text. The Google-Cloud-Vision API is being used to convert the OCR images to text. Once we have all the inputs in text format, we are then converting that given text into music and a visual image. For the conversion of text to image, we came up with a python algorithm that can google search the given text and display the image search result and for the conversion of text to music we have taken the help of an API called DITTY that gracefully converts any given text to music. The supporting website's frontend is built with react and backend is built with flask. The backend is running on Google-Cloud and frontend is hosted on Domain.com.
Challenges we ran into
We came across many challenges while building this product. Firstly, we had planned to generate images from text using Generative Adversarial Networks. But due to the high time and space complexity, we could train only on a limited dataset which could not provide the desired results. Hence we had to come up with another algorithm to serve the purpose. Converting text to creative musical tunes was also very challenging.
Accomplishments that we're proud of
We are proud of the fact that we have managed to complete the proof of concept of the entire project and proper implementation of about 75% of the project in less than 36 hours. This project can prove to be of great help for the social good of people with dyslexia.
What we learned
While doing this project, not only did we improve our technical skills but also learned a lot about dyslexia and how we can help them. We learnt how to work with various APIs, A great amount of troubleshooting and quite a bit about GANs and their implementation.
What's next for celeAI.tech
We plan to work on our GAN model with better datasets and arrive at more accurate and creative results. Some research told us that dyslexic people will be able to read and interpret better if the words/letters are placed at certain angles. We would love to try and implement this on our website.