Renaissance Art Style Transfer

Testing image
The Great Wave off Kanagawa's artistic style applied
The Great Wave off Kanagawa
Starry Night's artistic style applied
Starry Night
The Scream's artistic style applied
The Scream

Inspiration

The inspiration for this idea came from using Snapchat and applying one of their various different filters onto my face. While most filters simply overlay effects onto your face there are a few that completely change the way your face is seen through the camera. A style of some other image is applied to the contents of your face and that got me thinking about a way to convert photographs into paintings.

What it does

The Python script makes use of Convolutional Neural Nets to transfer the style of one image (in this case our desired painting) onto another image (in this case the picture we want to turn into a painting).

How We built it

This project is an implementation of a popular paper https://arxiv.org/abs/1508.06576. The program uses neural representations to separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic images. Convolutional Neural Nets are used to apply the stylistic elements of one image onto the other and this is done by focusing on two metrics: content of the image and style of the image. The goal was to minimize both content and style loss when transposing the two images. Content of the image is captured by collecting the arrangement of pixels while the style of the image is interpreted through capturing texture information. We wrote a Keras Deep learning model that uses image classification to capture content and uses convolutional neural nets to capture the style. The pre-trained VGG16 model from the Keras library was used for image classification and convolutional neural nets trained for image classification already knew how to encode perceptual and semantic information about the image. Those learned semantic information was used for the style representation.

Challenges I ran into

The biggest challenge developing this was the lack of access to computational power. Training Machine Learning models is a computationally expensive task and this one was no different. It took over 2 hours to complete only 10 iterations of loss minimization. This made testing almost impossible because only one iteration could be tested at most and that produced very noisy pictures. We relied heavily on the paper cited in the section above to structure the code and left testing till the very end.