Inspiration
We came with a simple idea of developing a typing game that scores user based on how fast they type a given text. After some thinking of what can we do with that data, we figured an interesting solution to a serious problem. Apply the data to solve the problems of screen readers that visually impaired people use.
What it does
There are two parts to the project:
- We present the users with a game that scores them based on their accuracy of typing. The players get a score, the time they took and the mistakes they made. The sentences contain markers around the words that have emphasis. After playing for a while, the player gets accustomed and learns where to place markers. Then we start providing plain sentences and users add emphasis markers.
- We use this data to build a machine learning model. To create the initial model, we give the players pre-marked sentences to mark. Using their markings, we generate a model to build their profile. For eg players marking plain sentences correctly will receive much higher XP as compared what they would get playing just the typing game. These profiles will be used to predict which is the correctly emphasized sentence. The correct sentence will be provided to a text to speech service that understands the emphasis markers and produce a prosodically pleasing speech. The screen reader service is hosted on Azure Cloud. It takes the .wav files and the correctly emphasised sentence and sends the .wav file as the output of TTS.
How I built it
The mobile app is developed for iOS using XCode and Swift. The initial, unmarked sentences provided to the players are manually marked to generate a classification model using perceptron. Now the players are provided with completely unknown sentences and using the model we classify the marked sentences into correct or incorrect. The correct sentences are provided to a TTS service that analyzes the emphasis marks and generates a .wav file which is later used on the online screen reader.
Challenges I ran into
iOS development was really new to our teammates. Getting a good TTS service that incorporates prosody was challenging, Microsoft mentor Tim really helped with this.
Accomplishments that I'm proud of
Finishing a working model of the product, integrating various services
What I learned
Technical skills: iOS, Azure Web deployment. Planning really helps as does taking breaks.
What's next for Fastype: Play for humanity
Improve the machine learning model, get more data, introduce new levels in the game, make it multiplayer, possibility of monetization by providing the audio to various audio books developers
Log in or sign up for Devpost to join the conversation.