Like the good old days, Windows Media Player has its moving screensaver when we play a music file with it. This project is inspired to create a modern version of that screensaver by using AI, Deep Learning and some Image Processing. The project serves to helps musicians by generating and providing an artistic video for them to compile together with their music.
What it does
This repository helps users to perform neural style transfer on video files easily. The steps are simple enough that the users can freely choose their own custom input video files.
How we built it
The core technology used for this project is PyTorch by Facebook Research and OpenCV(a popular open source library for computer vision).
PyTorch is used as the main deep learning framework. It is used for model training and inference. It is very useful when you can dynamically assign the tensors to either GPU or CPU for computation. The tutorials on PyTorch is also abundant and the open source community is strong.
OpenCV is the main library used for image processing. We used it to read the images and writing them into videos. We also use it to create transitions for our video.
Challenges we ran into
- When starting out, I only know how to do neural style transfer on a single image by following the tutorial by Udacity PyTorch github repo. Needed to search on how to do video neural style transfer using PyTorch.
- Having a output expectation in mind, the final output is not favorable because the way style transfer works in the Udacity tutorial is not same as the way in the repo we found on github. So, we discard the method shown in the tutorial.
- We needed to create a image blending effect for my transitions. In the end, I found out that OpenCV has a function that I can hack on to create the image blending effect.
Accomplishments that we're proud of
We have used deep learning and AI in the field of art and creativity! Where in most cases, deep learning is used for industrial advancement for example defect detection, ROI segmentation. Deep learning and AI has changed the norms and legacy ways of how we perceive problems. Art and creativity should not be left out, AI will also inspire current generation of artists and bloom out new ideas.
What we learned
- How to use PyTorch.
- How to do Neural Style Transfer on a single image.
- How to do Neural Style Transfer on a video.
- Time management.
What's next for Music Video Generator with Pytorch
- GUI for the application.
- Make it more robust where the style will change according to music volume, and beats.
- Apply Neural Style Transfer on audio signals instead of images. E.g. transfer a person's singing style to another.