• 

Text-to-image neural network

• Linus Sun, lsun23
• Nishka Pant, npant1
• Jay Sarva, jsarva

Introduction: What problem are you trying to solve and why?

If you are implementing an existing paper, describe the paper’s objectives and why you chose this paper.

https://arxiv.org/pdf/1605.05396.pdf generative adversarial network that generate plausible images of birds and flowers from detailed text descriptions. This is a data generation problem.

Related Work:

https://arxiv.org/pdf/1612.03242v2.pdf https://towardsdatascience.com/text-to-image-a3b201b003ae Summarizing the blog post, the text is embedded into a feature space. That embedding is then concatenated with random noise and fed into the generator network. The text embedding is also concatenated onto the image output from the generator, and that is fed into the descriminator. The parameters of both the generator and the descriminator are updated with gradient descent until the descriminator predicts that the image is a member of the training set with approximately 50% accuracy. The Zhang et al. paper introduces the idea of solving this task with a StackGAN, which is used to generate larger images in higher resolution. The StackGAN uses a two stage process, where the first stage generates a low-resolution image with rough sketches and basic colors, and the second stage generates high resolution images with photo realistic details.

Previous implementations of the project:

https://github.com/hanzhanggit/StackGAN

Data: What data are you using (if any)?

If you’re using a standard dataset (e.g. MNIST), you can just mention that briefly. Otherwise, say something more about where your data come from (especially if there’s anything interesting about how you will gather it).

How big is it? Will you need to do significant preprocessing?

Captioning datasets from : https://cocodataset.org/#download

Methodology: What is the architecture of your model?

Stacked GAN, Stage-I and Stage-II GANs; Stage-I outlines primitive shape and colors of object, Stage-II provides details Stage-I = embedding, generator,

How are you training the model?

train a stacked deep convolutional generative adversarial network encoded by a hybrid character-level convolutional recurrent neural network use feed-forward inference conditioned on the text feature

If you are implementing an existing paper, detail what you think will be the hardest part about implementing the model here.

The generative network will be the most difficult, as we have not really implemented anything like it in class; it will be a new experience.

If you are doing something new, justify your design. Also note some backup ideas you may have to experiment with if you run into issues.

N/A

Metrics: What constitutes “success?”

What experiments do you plan to run?

We will perform primarily qualitative analysis using the images generated from our network and comparing them to the training dataset of images.

For most of our assignments, we have looked at the accuracy of the model. Does the notion of “accuracy” apply for your project, or is some other metric more appropriate?

The notion of accuracy applies only in the sense of how ‘good’ the image looks to the human eye. We are basing success on how realistic the output images look, and whether or not they match the descriptions used to produce them.

If you are implementing an existing project, detail what the authors of that paper were hoping to find and how they quantified the results of their model.

The authors were trying to have images created that were related to the caption presented. Results were quantified by comparing human rankings with other similar generative models. The authors performed primarily qualitative analysis, comparing the training image database, their GAN baseline, GAN-CLS with imagetext matching discriminator (subsection 4.2), GAN-INT learned with text manifold interpolation (subsection 4.3) and GAN-INT-CLS which combines both. They also used AlignDRAW to perform their qualitative analysis.

If you are doing something new, explain how you will assess your model’s performance.

What are your base, target, and stretch goals?

Base goal: Our network can produce images that clearly outline the given object we are attempting to describe. Target goal: Our network can produce images that clearly outline the given object we are attempting to describe and also contains details specific to that object (eg. the color of a certain bird species' tail). Stretch goal: Our network can produce images that clearly outline the given object we are attempting to describe and also contains details specific to that object, as well as generating a realistic background for the image containing the object.

Ethics: Choose 2 of the following bullet points to discuss; not all questions will be relevant to all projects so try to pick questions where there’s interesting engagement with your project. (Remember that there’s not necessarily an ethical/unethical binary; rather, we want to encourage you to think critically about your problem setup.)

What broader societal issues are relevant to your chosen problem space?

This sort of model could be used to manufacture deepfakes, essentially generating images that may look real but are just generated from a caption (and this image may in some way be used to misinform the public).

Why is Deep Learning a good approach to this problem?

Deep Learning is a good approach to this problem because a systematic approach to generating images from captions would likely either be very limited, if selected from a prebuilt set of image assets, or perform very poorly, if performed by generating an image pixel-by-pixel. In contrast, a network that learns how to generate an image given a caption will perform much better and be very flexible in the range of images it can produce.

What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain?

Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?

How are you planning to quantify or measure error or success? What implications does your quantification have?

Add your own: if there is an issue about your algorithm you would like to discuss or explain further, feel free to do so.

Division of labor: Briefly outline who will be responsible for which part(s) of the project.

Generator - Jay Discriminator - Linus Data collection - Nishka

First Check In

Introduction:

We are reimplementing the 2016 ICML paper Generative Adversarial Text to Image Synthesis. With this project, we are aiming to use text descriptions to generate realistic looking visuals that reflect the descriptions accurately. If we have time, we are also planning on implementing an architecture that can be found in the 2017 ICCV paper StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. This second paper is an improvement on the results generated by the first paper, with images that look more photorealistic with more details and sharper edges.

Challenges: What has been the hardest part of the project you’ve encountered so far?

So far, our greatest challenge has been dealing with data collection and the preprocessing of our images/captions. We went through the process of tokenizing/embedding our caption data so we could process it appropriately.

Insights: Are there any concrete results you can show at this point?

We have not yet managed to get to a point where we can produce concrete results. We are still in the process of writing the code for generation and visualization of results.

How is your model performing compared with expectations?

Since we have not yet reached a point where we can visualize the results, we are unable to see how our model performs as compared to our expectations. The benchmark which we are using to determine how well our model is performing is purely qualitative. The notion of accuracy applies only in the sense of how ‘good’ the image looks to the human eye. We are basing success on how realistic the output images look, and whether or not they match the descriptions used to produce them.

Plan: Are you on track with your project?

We are on track with our project; we will be able to finish our architecture within the next 9 days so that our project is ready by Deep Learning Day.

What do you need to dedicate more time to? We definitely still need to dedicate more time to finishing the architecture design. Past that, there will be more time spent tweaking the project in order to try and make our results better.

What are you thinking of changing, if anything? We are currently not thinking of changing anything about our current planned implementation between now and the deadline.

Final Report:

Text-to-image neural network

• Linus Sun, lsun23
• Nishka Pant, npant1
• Jay Sarva, jsarva

Introduction:

We are reimplementing the 2016 ICML paper Generative Adversarial Text to Image Synthesis. With this project, we are aiming to use text descriptions to generate realistic looking visuals that reflect the descriptions accurately. We use the Caltech-UCSD Birds-200-2011 dataset, with 11,788 images of birds as the training dataset. The 'captions' for each image involve a tokenized list of characteristics about each bird which are obtained via the 'classes' that the different bird images are separated into. We implement a generator and discriminator model with architectures based on the aforementioned paper.

Methodology:

We implemented this project using a Generative Adversarial Network (GAN) with the . Our GAN consists of one generator network and one discriminative network. Our generator network uses the Adam optimizer with a learning rate of 0.0002. It consists of a text embedding component with a ReLU activation and a deconvolutional component consisting of a linear layer and 3 transposed convolution (deconvolution) layers alternated with batch normalization. To run the generative network, a given caption is embedded and concatenated to a representation of our network's latent space; this concatenation is passed into the deconvolutional network and output as our generated image.

Our discriminative network uses the Adam optimizer, also with a learning rate of 0.0002. It consists of a text embedding component with a leaky ReLU activation, a convolutional component consisting of 3 convolution layers alternated with batch normalization, and a fully connected component consisting of a convolution layer, a batch normalization, a flatten later, and a linear layer. To run the discriminative network, an image is passed into a convolutional component, its corresponding caption is passed into a text embedding component; these are concatenated and run through the fully connected component, which outputs a score between 0-1. This score represents the confidence the discriminative network has that the given image is real and not created by the generative network. We want this score to converge to 0.5, as we will pass in an equal amount of real and generated images.

Results:

When we train with batch size of 100 over 200 epochs, our model generates images that have the general outline of birds, but the images lack detail. This is related to both the low number of epochs and the low quality of each image (64x64 pixels). Many of the images appear to have the outlines/characteristics of birds, yet the backgrounds are not very realistic, often never looking like any real-life background. These would likely improve with higher epochs, which were not used due to lack of computing power.

Challenges:

We ran into challenges while trying to decipher what the paper was describing with regards to some portions of the model architecture, as well as the equations for the loss. We spent more time reading tangential literature about deconvolutional networks and GAN implementations to overcome these hurdles.

We also changed datasets from our original plan, because the MSCOCO dataset (which is what we originally intending to use) was too large for the compute we had access to. We changed to an updated version of the dataset provided in the paper.

Reflection:

We feel that the project turned out well! We met our target goals, we have produced images that outline the shapes we intended in the captions. These images appear to contain the intended features.

On a very basic level, the model does work the way we were expecting it to. The images produced do look like low-resolution versions of birds with the features that are specified in the caption. Our approach didn't change very much over time, it just took us longer than anticipated to work through our plan. We ran into some challenges with getting our initial baseline working, so we didn't have as much time to implement the more complicated architectures that we intended to implement. The paper we were re-implementing was not specific about many aspects of the model itself (ex. not specifying anything about the deconvolutional layers of the generator), so we had to construct these parts ourselves. Given that we are not very familiar with GANs, this was mainly guesswork. The model took hours to train, so we had limited opportunity to train different versions of the model to compare results. If we had more time, we could probably put more effort into playing around with the architecture and tuning our hyperparameters in order to create results that look more photorealistic.

From this project, we learned about generative adversarial networks and how to implement the architectures from research papers. We also had a bit of practice with hyperparameter tuning. We wrote the entire training pipeline from scratch, so we learned how to construct entire DL based projects independently (without autograder support :) ).

Built With

• python
• tensorflow