The primary motive for developing this solution, was the feeling of empathy for my neighbors children. I wanted to assist her working mother in educating her child. This application, in its nascent stage, engages kids and propels them to follow the strokes to paint wonders. Being heavily inspired by Google AI Experiments, we were determined to use Sketch RNN in this endeavor.
What it does
This is a stand-alone web application, capable of entertaining children, in simple yet impactful way. The child would be given a chance to draw a self-selected or randomly generated word on the screen, and would be assisted with our AI assistant. The app will be with your kid throughout in the drawing process, when the kid will be stuck it will draw the next strokes to be drawn slowly to complete the drawing. The kid can then follow the app and try to complete the drawing. This way teaching the kid to draw. The parents need not even be with them, the speech assistant tries to engage the kids and encourages them to try and draw when they are stuck. If they want, the guardians can sit with their kids and teach them alphabets which make the spelling of the word they are drawing. The web app also offers a functionality of a "hint" image to show the kid how a certain object looks.
How we built it
We divided the job into four main parts -
- Understanding and tweaking the code for Sketch RNN Demos available on Github, which completes a drawing once a user just draws one stroke on the screen. It also keeps on giving different possible drawings of the same object once it completes the drawing once, until the user changes the selection of the object to be drawn. Our code tweaks the implementation such that it behaves in an assistive manner where it only helps the kid when he/she is stuck by providing the next possible stroke. Also, we provide a stop button functionality which stops the generation of continuous generation of drawings.
- Using the Google Cloud Text-to-Speech API - We used this API to provide a speech assistant functionality that notifies and encourages the kids constantly whenever they select a new object or are stuck at a point and are unable to draw.
- Using the Google Cloud Translate API - Our app speaks and displays text in four different languages - English, German, French and Spanish.
- Using Google's Quick Draw API for programmatically extracting images of objects from all categories to provide the hint functionality to the users. In the hint functionality, the user can see how a certain object's drawing would look like.
Challenges we ran into
As first time hackers, all of the above steps were challenging, but the biggest challenge we faced was to translate the contents of the entire page without reloading the entire page again. We used the session cache information to detect the current language and the change in the language to solve this problem.
Accomplishments that we're proud of
We are proud that we took the plunge of coming to our first hackathon ! We realized that we were capable of much more that we had thought so. We are proud that we got a general understanding of using Google Cloud APIs and used two of the most popular ones in our project. We are proud that we could understand a complex code which was written in a language we were unfamiliar with before we came to this hackathon and still managed to make the necessary tweaks in the program. And finally, we are proud that we made a project that is cool, useful, teaches kids and makes life's of working parents easier!
What we learned
We learned using Google Cloud APIs, learnt reading complex code and tweaking it and most importantly we learnt team work and not loosing hope when nothing's working out!
- Kids ages 2 years to 4 years
- Parents with kids.
- Kindergarten teachers.
What's next for Teach To Draw!
We want to make the Sketch RNN model better in drawing the appropriate images. We also integrate even more languages in text as well as speech. Refine the app's UI and focus on its usability, We want to be able to build a system which converts a textual story to an animated one automatically