Inspiration

Prisma beats the world’s filters, but its functionalities are only limited to processing still images. We want to make the video artistic as well, so we create Re:Art.

What it does

Re:Art is a Web App that aims to transform a video into an animated artwork by furnishing it with a particular style or pattern of paint chosen by the user.

How I built it

Back-end: By utilizing the open-source VGG19 neural network model and Torch Framework, We set up a comprehensive back-end still image stylization service based on the AWS and G2 instance in the EC2 service. We achieve the non-linear video processing result through Open CV (Open Source Computer Vision), which enable the web app Re:Art allowing users to process a self-designed animated artwork.

FlowChart

Read user input: When a user uploads a video and chooses a picture style, we will read in the video and style they choose, and we will decide the frame number based on how long we want the output gif file to be.

Transform the video: We pick up the images based on the frame rate, and then crop the images into the size with the ratio 8.5:11, which is roughly 300px * 388px. Also, we uses advanced compression algorithm to make the image further smaller without losing much of its original quality. Due to the undesirable efficiency of the Neural Network Framework, we came up with several ways to increase our efficiency use multi-parallel subprocess to transform the image simultaneously, and use the resources of the server to its maximum. Because we depend on GPU rather than CPU to have a better performance, we chose the AWS instance that has more GPUs. Because the feature of the Open Source Torch Framework is based on Lua, we manipulate the iteration time on each image to achieve an effect of transition, giving user a feeling of more dynamic stylized images.

Output Gif: We store the image to an array and restructure them to form a gif, which is finally presented to the user.

Front-end: We designed a user-friendly Web App using HTML5, CSS, JavaScript, JQuery to orient the users to upload their videos, choose a style that they prefer and follow the page to finally get the stylized gif. On our home page, we designed as few icons and concise instructions as possible to make it clear to the new users on how to use the app. After uploading a video, the user can choose within a scroll bar several different styles of artworks to merge the video with. Then our app throws the job to the server to complete stylization process. When it’s done, the user is directed to another page where he/she can view the resulting gif and share it to Facebook, Twitter, Google Plus etc.

Challenges I ran into

When we set up the service in the beginning, we need to install a bunch of libraries. The fatal thing is that these libraries have version conflicts, which took us a long time and drained our energy. At first we decide to se TensorFlow, an Open Source Software Library for Machine Intelligence, but the time this library take to process video is too long. Thus, we have to change our Framework to Torch. We worked with several programming languages (Lua) and Framework (Torch) that we’ve never encountered before, so we had a high learning curve. The stylized image was rotated after the transformation, and we need to align it to normal. The biggest challenge is: it got super hard to code on 6am!

Accomplishments that I'm proud of

We achieve the functionality of not only stylizing the static image, but also for video. We see our web app works pretty well.

What I learned

Sleeping is reeeeeeeaaaaaaally important.

What's next for Re:Art

Migrate the app to multi platforms. Immersed in the conversation with our comprehensive services.

Share this project:
×

Updates