Title: Fashion Forward - Regenerating Clothing Image with Different Attributes

Github: https://github.com/yangzheng-brown/CSCI2470-Final-Project

Presentation Slides: https://docs.google.com/presentation/d/1CsNlzR7zpM9r3y_vJl9gy2-UgylSWrfdfHeRa0zPBe0/edit?usp=sharing

Final Writeup/Reflection: https://docs.google.com/document/d/1X6juBGS5hCCWxgOtodd6hfH0U-bvKpmfHWszvBWTv4U/edit?usp=sharing

Developers

Juefei Chen (jchen385)
Yurui Zhang (yzhan636)
Yang Zheng (yzhen102)

Introduction

Personalization has emerged as a significant trend in recent years, with more consumers seeking unique and customized clothing options that reflect their personalities and preferences. However, creating a wide variety of designs can be time-consuming and costly for fashion retailers. Our main objective of project is to create a deep learning model that can regenerate a given clothing image into a new image with different attributes and clothing type. For instance, if we have a hoodie with color block, we can instruct the model to transform this image into a jacket with a palm pattern. We hope this deep learning model could enable retailers to provide more personalized and customized clothing options for their customers, thereby enhancing their shopping experience.

Related Work

The Text2Human paper, authored by Bokai Sun, Junjian Chen, and Wenyong Huang, is relevant prior work to our project. The paper proposes a method for generating human images from textual descriptions, using separate neural networks to process text and image features, and combining them to generate the final image. The authors also propose methods to improve control and diversity in generated images, such as using attention mechanisms to adjust image features and introducing noise vectors for added randomness. The paper provides a promising approach to generating human images with greater control and variety. URL: Text2Human: Text-Driven Controllable Human Image Generation (arxiv.org)

Data

We will use a clothing dataset called DeepFashion, which contains over 800,000 images, covering a diverse range of clothing items, including well-posed shop images to unconstrained consumer photos. It spans across 50 different clothing categories and has 1,000 attributes. Each image is annotated with a bounding boxand clothing type, making it easier for our model to learn and recognize various clothing types. The dataset can be found in https://drive.google.com/drive/folders/0B7EVK8r0v71pQ2FuZ0k0QnhBQnc?resourcekey=0-NWldFxSChFuCpK4nzAIGsg

Methodology

We will choose the Variational Autoencoder (VAE) to build our model, where the encoder is a convolutional neural network (CNN) to process the input clothing image and learns the Gaussian distribution that represents the latent space; the decoder is another CNN that takes a sample from the latent space, conditioned on the input attributes (e.g., clothing type, pattern), and reconstructs the transformed clothing image.

We will train our VAE model with the dataset in the previous section by firstly splitting the dataset into training and test sets. During each training iteration, we will pass the input image and the desired output attributes through the encoder to obtain the latent space representation. Then we sample from the latent distribution and condition it on the desired output attributes, and pass the sampled and conditioned latent vector through the decoder to generate the transformed clothing image.

We will optimize the model using the loss function mentioned in the next section, with adjustment of the model's hyperparameters. And using reconstruction error as a metric to measure how well the model can reconstruct the input images.

If we encounter issues with the VAE architecture, we can modify the model’s architecture by adding additional layers or changing the activation functions, and we can try different loss functions or loss function weights. We can also try GANs and Diffusion architectures.

Loss & Metrics

To training the model, we propose a weighted combination of loss function consisting with reconstruction loss (L1) and the regularization loss (L2). Reconstruction loss measures the difference between the original input data and the reconstructed data generated. The reconstruction loss is the mean squared error between the original input data and the reconstructed data. The regularization loss encourages the model to learn a smooth and continuous latent space by penalizing the deviation of the learned distribution from a prior distribution, usually a multivariate Gaussian distribution. The regularization loss is the Kullback-Leibler (KL) divergence between the learned distribution and the prior distribution. Therefore, the overall loss function is the sum of these two terms: Loss = Reconstruction Loss (L1) + ɑ * Regularization Loss (L2) where ɑ is a hyperparameter that controls the relative importance of the reconstruction and regularization losses.

To learn how well the model performs in generating new images, we propose two evaluation metrics, reconstruction error, and attribution prediction accuracy. Reconstruction error measures how well the model can reconstruct the original input. A lower reconstruction error indicates that the generated images are more similar to real images in terms of their visual features, such as color and texture. For attribution prediction accuracy, a pre-trained predictor is employed to predict the texture attributes of the generated images, and the accuracy of these predictions is reported as a measure of the realism of the generated texture.

The base goal of the project is to implement and train the VAE model, and the model should be able to generate a set of new image. The target goal is to improve the model to generate new images with better quality and accuracy. The stretch goal is to develop a font-end page for users to input prompts and generate new images accordingly on the page.

Ethics

The following two ethical issues need to be discussed for our project:

For the dataset, are there any underlying historical or societal biases might it contain?
The dataset is collected from well-posed shop images and consumer photos. But it may contain biases from the fashion industry, such as the thin body shapes, cultural clothing styles, or seasonal trends. These biases may result in the reinforcement of existing stereotypes and certain fashion norms, and limit the diversity of clothing options.
Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?
The major “stakeholders” are designers and customers. If the algorithm fails to accurately generate images, customers may be disappointed with the results, leading to a poor shopping experience and potential loss of sales.

Division of Labor

Juefei, Yurui and Yang will help each other with every part of the project. However, each of us will take a leading responsibility for certain parts. Yurui will focus on preprocessing and building the structure of the model, Juefei will focus on the implementation of encoder block and ethical issues discussion, and Yang will focus on the implementation of decoder block and poster designing. We will then combine each work together into the final model.