Implementing rNVP to learn distributions

Deep Learning Day Poster

Title: Implementing rNVP to learn distributions

Who:

Diana Manzano - dmanzan1;
An Vo - avo2;
Paul Muser - pmuser;
Shaun Kohli - skohli5

Introduction:

What problem are we trying to solve?

We look to solve the problem of unsupervised learning of a probabilistic model via real-value non-volume preserving (rNVP) transformations. Based on the change of variable formulas for functions of random variables, one can transform one r.v. to a different r.v. if the determinant of the Jacobian of said transformation is computable. For many transformations, this process is either impossible analytically, or very expensive numerically. rNVP are invertible, stable mappings between a data distribution and a base distribution (Multivariate Normal). Said mappings also have Jacobian determinants that are easily computable. This results in a generative system that can learn and produce (relatively) exact and (relatively) efficient log-likelihood, sampling and inference of the learned distribution. The base distribution being Gaussian is also interpretable from the outcome of the learning process. Related work: https://arxiv.org/pdf/1605.08803.pdf

Data:

Our initial target goal is to feed rNVPs data samples from Mixed gaussian distributions with known parameters so that the rNVP can learn the bijective transformations required to transform one Multivariate Normal into an approximation of the distributions from which the data is sampled. While we aren’t using a standard dataset, we are fortunately able to avoid any preprocessing work because we will be generating our dataset directly and can thus insure its format is consistent with what we want to feed into the model.
Our training/testing data will have two components. The first is a list of sampled data points and the second is our label which consists of the ground-truth parameters of the distribution from which we sampled. We will construct thousands of different distributions by varying parameters and will sample thousands of data points from each of these distributions.
For our stretch goal, we will be trying to use rNVPs in conjunction with texture network models. In order to train this we will be using the same data set used by Ulyanov et al in their paper on texture generation (IMAGENET).

Methodology:

How are you training the model?

We will use the different inputs described above for the different levels of goals, and will be feeding these inputs into a model consisting of (essentially) a chain of RNNs. We plan on utilizing the cloud computing resources provided to train and run our model in a more efficient environment.

If you are implementing an existing paper, detail what you think will be the hardest part about implementing the model here.

There are a number of challenges in implementing the rNVP paper. We think scaling the architecture and implementing batch normalisation will be the most challenging part. On a bigger-picture scale, making sure we all have a good grasp on the underlying math involved is on one hand challenging but also extremely important to us being able to adequately implement the paper.

Metrics:

What experiments do you plan to run?

Base goal: Our base goal is to test our rNVP model on mixed gaussian distributions with known parameters.
Target/Stretch Goal: Our target/stretch goal is to use this model to create a simple texture generative model with large amounts of variability and entropy in outputted samples.

Does the notion of “accuracy” apply for your project, or is some other metric more appropriate?

Base goal: Accuracy (as it’s been defined thus far) isn’t applicable in this case since we are using DL to create invertible, stable mappings between data distributions and base distributions. Instead we can compare the distributions using another metric. Since these distributions have known parameters we can use KL to compare the similarity of the output of the DL network to the true distribution.
Target/Stretch Goal: We do not have a notion of “accuracy” for this generative model. The Ulyanov paper details a method for calculating loss in the task of texture generation that we can use as our loss in our neural network, but doesn’t give a measure of accuracy.

If you are doing something new, explain how you will assess your model’s performance.

Since these distributions have known parameters we will use Kullback–Leibler divergence to compare the similarity of the output of the DL network to the true distribution and a KL divergence score close to zero will be seen as a success as that indicates that our DL network functions well and has learnt how to model the distributions.
Target/Stretch Goal: We will use the loss calculation from the Ulyanov paper that will be linked above to train our rNVP model. Success will indicate a low calculated loss value as well as visual similarity between the texture inputted and the outputs (as well as variation in the outputs).

What are your base, target and stretch goals?

Base goal: Our base goal is to have a functioning rNVP model that works on mixed gaussian distributions with known parameters.
Target Goal: Create a simple texture generative model that takes in textures and outputs images that broadly look similar.
Stretch Goal: Create a texture generative model that takes in textures and outputs images that could be a real extension of the original image.

Ethics:

Why is Deep Learning a good approach to this problem?

In order to adequately answer our questions (base, target, and stretch), we require a very large input dataset. It will be at a scale unmanageable to work with without abstracting and parallelizing the calculations. So, using a deep learning model is a great way to get past the complexity of the task at hand.. This is a topic of ethical concern because it would be irresponsible to expend the considerable resources required to repeatedly train as we adjust hyperparameters and finally use the deep learning model if there was a simpler and more resource-efficient way to complete our task.

What broader societal issues are relevant to your chosen problem space?

It is important to recognize the implications of the fact that we are producing a generative model. There are alternate applications beyond just texture generation that this model could be adjusted to work for. For example, one of the papers we looked at regarding Applied MEFNs used this type of model to generate novel images of faces based on a dataset of celebrity faces. The ability to generate false faces (or any similar type of false imagery), if convincing enough, could have negative consequences if implemented by people who wish to do harm.

Division of labor:

Our original plan was to split into two groups and divy up these four tasks:

Implement model (first pass)
Implement scaling + batch normalisation
Training + parameter tuning + testing
Post-testing stuff (posterior predictives, KL, graph-making, etc. etc.)

In reality, we ended up needed to help each other a lot with conceptually understanding the model architecture and debugging our code, so we often just were all on a call or messaging each other while working on closely related parts of the project. Everyone contributed to the code and final report. As for a few distinctly more individual efforts:

An recorded the audio for the presentation video, and helped all other team members conceptually grasp the model structure and probabilistic background
Diana put together and published the video
Diana and Paul worked on designing and filling in the poster
Shaun was a major contributor to the debugging and organization of our code, helping produce some of the most well-commented code we've all seen in a while