Traditional Chinese Ink Neural Style Transfer

Deep Learning Final Project Outlines

Team Name: Smart Ink

Team Members: Zhangyi Shen (zshen5), Yuanfeng Li (yli586), Tengfei Jiang (tfjiang31)

Github Repo: https://github.com/TJiangF/Smart_Ink_2022

Google Drive Link: Our reports, video presentation, and posters are submitted to https://drive.google.com/drive/folders/17U6Dvc3YSnUEntqRiQaSWbggMXpHeKBF?usp=sharing

Summary

We chose to work with style transfer and experiment with the Chinese ink style, which is challenging since it is more like freehand brushwork. We plan to exploit CNN for neural style transfer and GAN for image colorization.

Introduction

We first thought of exploring neural style transfer within the context of cartooning images; however, we realized that many projects have already explored this area and most neural style transfer projects focus on western styles. As three Chinese painting lovers, we decided to try out neural style transfer with the traditional Chinese ink painting style. We will implement additional methods and constraints to make neural style transfer more compatible with Chinese painting techniques such as blank-leaving, brush stroke, or ink washing.

Related Work

Core paper: Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks https://openaccess.thecvf.com/content_iccv_2017/html/Zhu_Unpaired_Image-To-Image_Translation_ICCV_2017_paper.html For tasks where paired training data are not available, the paper talked about an approach Cycle GAN to translate an image from a source domain X to a target domain Y without paired examples.

ChipGAN: A Generative Adversarial Network for Chinese Ink Wash Painting Style Transfer https://dl.acm.org/doi/10.1145/3240508.3240655 The paper introduced a model ChipGAN based on CycleGAN and enforced three constraints specific to traditional Chinese ink painting styles including voids, brush strokes, and ink wash tone and diffusion.

Data

Our dataset “ChipPhi” comes from a paper named “ChipGAN: A Generative Adversarial Network for Chinese Ink Wash Painting Style Transfer” which consists of ink-wash painting photos of real scenes from the Internet and art studio. The horse dataset contains 1630 images of horses and 912 images of paintings by the famous horse painting expert Xu Beihong. The landscape dataset contains 1976 images of landscapes and 1542 images of paintings by the famous landscape painting expert Huang Binhong.

For data preprocessing, all the images will be reshaped to the same size. Since there are a limited number of real paintings, we plan to apply data augmentation techniques to the dataset such as flipping the images or adjusting the brightness of the images.

Methodology

After our investigation, there are two main branches for image style transfer with respect to deep learning, including CNN-based approaches and GAN-based approaches. Generally speaking, CNN-based approaches use CNN to extract style representations from the style image and then apply the style representations to the texture representation in the test image to get the result. However, for the traditional Chinese ink style, it is relatively hard to find a typical style image to represent “ink style” since different ink paintings have unique techniques, compositions, and painting styles. For GANs-based methods, its goal is to construct a domain mapping: X→Y, where X represents the input image domain and Y represents the “ink style” domain. This mapping is implemented by training a generator to generate the image, and a discriminator to tell whether it is an image from Y. The mapping is created successfully when both the discriminator and generator are good enough that the discriminator can tell the generated image belongs to Y. However, for a one-directional GAN, some image pairs (representing the same content in different styles) are needed. However, such image pairs are hard to obtain. To address this, Cycle GAN is a specified GAN structure that introduces a new inverse mapping F: Y → X on the basis of G: X → Y, and adds a consistency loss to the loss metric. To combine these two GANs, Cycle GAN eliminates the influence caused by the image content and thus ensures that the transformations are only related to style. Therefore, through Cycle GAN, we can just input two kinds of images: real images and some ink painting samples. They do not need to have the same content. We also do not need to worry about choosing a specified style “image”, since Cycle GAN can automatically extract all the images to an “ink domain”. Considering these advantages, we decided to use Cycle GAN as our basic network structure.

Considering that cycle GAN is a very general method, it may not be able to take into account some proprietary style features in ink painting. Therefore, we still need to investigate some more refined methods, such as some constraints or penalty terms, to obtain better ink painting effects. In short, we will first implement a vanilla cycle GAN structure, and later make targeted optimizations and adjustments to it to get the best ink style generator.

Metrics

We will compare the generated images with real paintings, and take basic aesthetics into consideration. Since the generated images are in an expected ink style, the features of the generated images, such as objects, backgrounds, and landscapes, will be compared to real paintings with similar contents. In specific, the details of the generated paintings matter a lot to realize the “ink stylization”. While the drawing techniques cannot be learned by the model directly, we have to evaluate the result with a combination of human judgments and similarity calculation to see if the generated paintings are more natural.

Ethics

We can raise ethical questions such as whether computer-generated art can be considered real art. On the one hand, computer-generated art simplifies the creation of art and learns from famous artists, on the other hand, if the algorithm is mature and advanced enough, it can be used to fake real paintings for illegal profits. Moreover, improper images may be converted to ink style as historical evidence.

Division of labor

Tengfei Jiang: Implementation of CycleGan Zhangyi Shen: Improvement in methodology Yuanfeng Li: Improvement in methodology & Evaluation of the generated paintings

All submission including the video presentation, the poster, and the final report are added to "Try it out" as a google drink link.