Picture Generation from hand drawn sketches

Inspiration

I use generative models (like VAEs or SWAE) in my research to simulate particle collisions in collider experiments. By reviewing the literature, I found the GauGAN paper, which described how one can generate real-looking pictures from semantic images (or hand-drawn images). Since then it was on my to-do list to code a GauGAN, but never found time to do it. The Hackathon was a great opportunity to start with this project.

What it does

Ideally: Given semantic images (or hand-drawn images) from streets, it is able to generate real-looking pictures from the drawing. In my case, the training is taking too long, so I wasn't able to finish the training. Because of that, there is no saved model available to demonstrate it on.

How we built it

I mainly followed the GauGAN paper and the repository of the paper. I used PyTorch and tried to train it with AWS sagemake.

Challenges we ran into

A lot. I started with coding and training a simple conditional GAN on a MNIST data set to review GANs. Another challenge was to understand the code in the GauGAN repository from the paper. I had a lot of technical problems. Since a lot of computational power is needed, I decided to run the training on AWS sagemaker. Because I had to change parts of the code, I decided to run the training on Google collab intead. During the training, the kernel died multiple times, and my screen froze after 3h of training.

Accomplishments that we're proud of

having a running GauGAN which need to be trained.

What we learned

• A much better understanding of GANs
• Understanding GauGANs better (but still not completely)
• learned how to use AWS sagemaker

What's next for Picture Generation from hand-drawn sketches

Training it on different data sets. Apply it for real-world applications and more industry relevant tasks, e.g. design

Built With

• python
• pytoch