Video Generation with GANs

Mason Zhang (mzhan104) and Jessie Li (jli277)

Initial Check-in

Introduction

Generative Video Model

After learning about image classification and image generation with GANs, we were interested in seeing how we could apply similar techniques to the multiple frames of images present in video. The paper we are re-implementing (https://arxiv.org/pdf/1609.02612.pdf) primarily seeks to study how generative video models can contribute to a deeper understanding of video recognition and classification tasks. The part that we are focusing on is the ability to generate a video from a static image, essentially generating the predicted future of an object in an image. Our goal will be to train a future prediction model on a seed image so that we can give our model a picture of a dog, and have it animate the doggo! This is useful because dogs are cute and will implicitly provide insight on the way computers learn to recognize motion. Ultimately, this project should create a model for unsupervised learning of object movement in videos.

Related Work

(https://github.com/zhangzjn/DTVNet)
- (https://arxiv.org/pdf/2008.04776v1.pdf)
- “DTVNet: Dynamic Time-lapse Video Generation via Single Still Image”
- This paper presents a framework to generate diversified time-lapse videos from a single landscape image. This work focuses on objects like clouds and fog which have less standard motion and utilizes two main submodules: an optical flow encoder (OFE) and a dynamic video generator (DVG). The OFE uses unsupervised methods to get the motion maps between consecutive images and encodes them into a normalized motion vector while the DVG learns about the motion and image contents and then constructs video frames respectively. The model follows the methodology of GANs, but encodes extra motion information when generating video to get more realistic results. The results summarized in the paper show that this framework was able to capture content information in higher quality and account for a more dynamic range of motion in comparison to results from MoCoGAN and MDGAN.
(https://arxiv.org/pdf/1903.04480.pdf) “Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks”
Original paper’s Github repo: (https://github.com/cvondrick/videogan)
- A similar project (https://github.com/geoff-penington/gifify)
Potential extensions:
- Including a sound sample (https://openaccess.thecvf.com/content_CVPRW_2019/papers/Sight%20and%20Sound/Yukitaka_Tsuchiya_Generating_Video_from_Single_Image_and_Sound_CVPRW_2019_paper.pdf)

Data

We are currently thinking about using one of the following datasets, focusing on the videos with dogs.

https://lmb.informatik.uni-freiburg.de/resources/datasets/StereoEgomotion/
http://calvin-vision.net/datasets/tigdog/ For preprocessing, we will need to transform the video into sequences of images/frames

Methodology

We’ll be mostly implementing this paper [http://www.cs.columbia.edu/~vondrick/tinyvideo/paper.pdf], which utilizes a generative image model that leverages adversarial learning. One network (the generator) creates a synthetic video and the discriminator will try to distinguish the generated videos with the real ones.

The hardest part will be implementing the separation of the foreground and background (so that the model only has to deal with motion in the foreground) and combining them later on.

After we implement the model and get a baseline on how it’ll perform for our dog tail-wagging task, we’ll try to improve upon it. Areas we’re looking into exploring are:

Changing the loss function so that it penalizes large translational shifts between frames of an image
Having a discriminator run on every single frame of an image to assess the doggy-ness of the frame
- Motivation: Original paper generates motion decently, but people get distorted heavily into blobs

More ideas will come up as we dive into implementation :) Furthermore, since this model will likely be very computationally expensive, we’ll need GPUs and will use GCP GPUs or Oscar.

Metrics

The authors of the paper created a baseline model. They asked workers on Amazon Mechanical Turk to determine which video was more realistic—the baseline or their model. It’s hard to judge the realisticness of a video. We’ll call it a success if the videos generated are not heavily distorted or blurry and portray semi-realistic motion (more specifically, if we use a dog dataset, making sure that they’re cute and don’t get distorted).

Base: Create a working generative video model that can create a 1second long 64x64 pixel video based on a still* image
Target: Have the generative video model work on an image of an animal with relatively little distortion
Stretch: Compare model performance with other notable frameworks in the are

Ethics

What broader societal issues are relevant to your chosen problem space? We’re mainly doing this to test the limits of neural networks in generating video, a task that’s been historically very challenging. However, realistic video generation has broad societal implications. Deepfakes are already emerging as a big problem, and that’s mostly grafting a target image onto an already existing video. Actual novel video generation without the need for a reference video would be even more dangerous. We’re a seemingly long way from that, however. This kind of generative video model also creates assumptions about the style and movement of the objects in the image, which can create broad generalizations into the way objects ‘should’ look and move.

What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain? Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm? Dataset was primarily collected from videos on YouTube by querying the name of the object. As such, it’s not necessarily representative of the way all animals of the same type move, but as the subjects aren’t directly human-related, there should be less obvious societal biases. Our project’s primary goal is to learn about and innovate in the video generation domain, not to solve a human-related problem. Since this model won’t ever be used in a human-related context, and since there are no stakeholders, there are almost no consequences if the model is not as accurate/successful as we want it to be.