FaceTransfer

GIF
Sample output -- any video/image combination works!

Inspiration

Instagram's custom user-made filters have allowed users to express their creativity in so many ways that haven't been possible before. I asked myself -- how can we take filters to the next level with machine learning?

What it does

First, you upload a still image of a face that you want to animate. It could be your face, your friend's face, a celebrity's face, even a drawing!

Second, you upload a video of a face talking or doing any expressions. It could be a lip-sync video from Instagram, a video of you talking, or any video you find interesting.

The app will then animate the still face image moving in the exact same way as the face video! You can the download and share this video.

The great part of this app is that you choose both the video and the image! There are no restrictions compared to other deepfake technologies.

How I built it

I used Swift to build the mobile app to demo the technology. The backend is a simple Python Flask server. I adapted the first-order-model repository (https://github.com/AliaksandrSiarohin/first-order-model) to animate the images.

Challenges I ran into

Working with videos/images in Swift is hard...also passing media from the frontend/backend and ensuring that it is consistent was a major challenge. For instance, on the backend, I realized that my videos were being automatically rotated into landscape when they were actually portrait due to a small autorotate feature of the library I was using, which was confusing at first.

Accomplishments that I'm proud of

It works!

What I learned

Machine learning is more than just research, and PyTorch is great because it allows you bring cutting-edge research to an actual product in no time at all.