Inspiration

What it does

It is a web application that converts rough sketches of faces into detailed and colourful cartoons.

How we built it

Dataset

We used the google cartoons dataset to train a Generative Adversarial Network that generates cartoon characters from rough sketches.

Preprocessing

In order to train the model so that it takes sketches as input we had to preprocess the dataset. We used Sobel edge detector to find the edges in the images, thus, producing a pseudo-sketches that seem to be good enough.

Training

For the trainig of the model we used Tensorflow and Keras. We trained the model on Google Colab, utilizing nVidia K80 GPU. We wanted to train it for 20 epochs, however, our Google Drive run out of space on the 8th epoch. Our model saved its progress on each epoch, but it failed to save the last one because of insufficient space. Luckily, we were able to restore its progress up to the 6th epoch.

UI

We built a basic webapp that uses tensorflow.js and fabric.js to build the canvases. We have also used the gridsystem of bootstrap.

Challenges we ran into

It was our first time playing with GAN. We took a short course on Udemy to understand the concept of it and then did a simple tutorial utilizing it. After that we were able to quickly get going and combine ideas from several sources to come up with this solution.

Another challenge we faced was converting the trained model to javascript tensorflow model and loading in into the browser. We had to convert the tensorflow model to keras and then json files.

Accomplishments that we're proud of

We managed to create and train an ANN model, load it into a web app and finally build and deploy this app for the limited time we had.

What we learned

We learned quite a lot about GANs and using machine learning models in the web.

What's next for Sketch to Cartoon

We need to further train the model for at least 25 epochs in order to get better results. We can also try to use another edge detection method or try and make the detected edge lines smoother using morphological transformations.