Prisma beats the world’s filters, but its functionalities are only limited to processing still images. We want to make the video artistic as well, so we create Re:Art.
What it does
Re:Art is a Web App that aims to transform a video into an animated artwork by furnishing it with a particular style or pattern of paint chosen by the user.
How I built it
Back-end: By utilizing the open-source VGG19 neural network model and Torch Framework, We set up a comprehensive back-end still image stylization service based on the AWS and G2 instance in the EC2 service. We achieve the non-linear video processing result through Open CV (Open Source Computer Vision), which enable the web app Re:Art allowing users to process a self-designed animated artwork.
Read user input: When a user uploads a video and chooses a picture style, we will read in the video and style they choose, and we will decide the frame number based on how long we want the output gif file to be.
Transform the video: We pick up the images based on the frame rate, and then crop the images into the size with the ratio 8.5:11, which is roughly 300px * 388px. Also, we uses advanced compression algorithm to make the image further smaller without losing much of its original quality. Due to the undesirable efficiency of the Neural Network Framework, we came up with several ways to increase our efficiency use multi-parallel subprocess to transform the image simultaneously, and use the resources of the server to its maximum. Because we depend on GPU rather than CPU to have a better performance, we chose the AWS instance that has more GPUs. Because the feature of the Open Source Torch Framework is based on Lua, we manipulate the iteration time on each image to achieve an effect of transition, giving user a feeling of more dynamic stylized images.
Output Gif: We store the image to an array and restructure them to form a gif, which is finally presented to the user.
Challenges I ran into
When we set up the service in the beginning, we need to install a bunch of libraries. The fatal thing is that these libraries have version conflicts, which took us a long time and drained our energy. At first we decide to se TensorFlow, an Open Source Software Library for Machine Intelligence, but the time this library take to process video is too long. Thus, we have to change our Framework to Torch. We worked with several programming languages (Lua) and Framework (Torch) that we’ve never encountered before, so we had a high learning curve. The stylized image was rotated after the transformation, and we need to align it to normal. The biggest challenge is: it got super hard to code on 6am!
Accomplishments that I'm proud of
We achieve the functionality of not only stylizing the static image, but also for video. We see our web app works pretty well.
What I learned
Sleeping is reeeeeeeaaaaaaally important.
What's next for Re:Art
Migrate the app to multi platforms. Immersed in the conversation with our comprehensive services.