MAD- Music, Art, Doodle

Home page
Cat Model
Basket Doodle
Map model
Flower Doodle
Bus doodle
Human Eye
play piano
contact

Inspiration

We were inspired to build something that could be used as a learning tool especially for the children and also fun. Seeing something visually makes our brain remember that thing for a longer time. So why not make visual doodles and figures for children to learn while having fun! Also one could spend their leisure time with their old school hobbies like art and music. This project may also help artists and graphic designers in their work to come up with newer patterns or ideas. Also its quite inspiring itself to make the use of neural networks in a fun way!

Inspiration for domain

deepdrawing= deeplearning + drawing
doodlemap=doodling + map( made by RNN model)
sketchme= sketching+ me

What it does

The deep learning algorithm mainly Sketch RNN tries to make the drawings of various objects or animals like "Cat", "Map", "Airplane", "Butterfly", etc. We need to select the tags and the artificial intelligence brain tries to make it and gives us a drawing related to what we had chosen. Children could use this tool to first learn the drawing by seeing it and then use the provided sketch pad to draw out on their own. Just like kids love to draw, they also love music, so they could also learn to play their very own mini piano! Not only kids, but adults can also use these tools in their leisure time to relive their old school hobbies.

How we built it

We built the project mainly using p5.js , ML5.js , Javascript , Tensorflow.js and for front end we have used HTML, CSS, and BOOTSTRAP.
This project is mainly based on Sketch RNN(recurrent neural networks) , which analyses human sketches and presents them on a vector format. RNN's can learn and remember context in prediction problems for machine learning. This is also the reason it has now been explored in modelling images.
The Sketch-RNN would seek and learn information from loads of drawings by humans which are again segregated into different classes and is trained to generate them.
The dataset for the neural network model was obtained from Google's QuickDraw(https://quickdraw.withgoogle.com/) , an AI web application where the researchers asked users to draw objects a specific class and https://github.com/googlecreativelab/quickdraw-dataset

Challenges we ran into

We needed to learn about LSTM and Sketch RNN and how to make it work as well as how the model could predict the specified object or tag. Loading of the models takes time but it eventually works fine.
Loading of 5-6 models of the same object so that on every Run Again , a new model would be loaded and drawn.

Accomplishments that we're proud of

We are proud that we could machine learning algorithm in a fun way to build something good and also deployed the application on the web to make them work.

What we learned

We learnt a lot about LSTM, RNN ( mainly Sketch RNN) and how to use these machine learning algorithms to make something fun and interesting! We also learnt that the sketch is represented as a set of pen stroke actions which again helps in establishing the vectors through the coordinates and how LSTM forms the encoder part of the RNN while Hyper-LSTM forms the decoder. When tested, the RNN performed with an accuracy of 80% to generate as well as distinguish the images.

What's next for MAD- Music, Art, Doodle

Although the accuracy obtained in the output drawing might not be that exceptional but still it may help the children to learn doodles or even help designers or artists to idealize patterns on a large scale. The sketch RNN might also be tried with variations with respect to encoder-decoder systems to improve the accuracy in the generated sketches. We hope to add more soothing features to the website to make it look more engaging and also make it more fun!

Slack: Join my Slack workspace! https://join.slack.com/t/techies-zk91923/shared_invite/zt-jg6sbdva-Jrh3dmM2KhnzC0zXGlWHkA