Rapid Drop, then Slow Improvement
Drag and Drop
Quality enhancement is a field that seems to have stagnated. Static mathematical methods have been assumed to be the best when, in this age of computing, static formulas may no longer be king.
What it does
Aper.io is a Machine Learning Web App that can takes in low quality web apps and makes the comprehensible. The application of deep learning within the field of quality enhancement has been largely unexplored, especially in the realm of video.
Given any low quality video, Aper.io utilizes a series of Deep Convolution Neural Network Style Transfer Models which, given a series of frames, is able to predict smooth interpolation images. Due to the fact that this application is a largely unexplored space, we trained our predictive style transfer models from scratch. We did our prototyping on our own local machines, and once we had a solid foundation, we moved all our training onto Google Collaborate, which was key in meeting the deadline.
How we built it
We were a little lost as to where to start on this process, so we just decided to see if there were any research papers on the topic. While there were no directly applicable papers, we did find a two particular papers whose ideas we found interesting and used as a starting point for our models and data processing.
From there we split up duties: some of us focused more on updating and debugging the models, and figuring the right architecture ( which involved a great deal of CNN tracing and implementing custom modules in Keras). Others focused on the backend of correctly combining the model output and the original video into a new higher resolution video and then smoothing out the abrupt insertions. While at the same time, others of us focused on developing a UI/UX that would get people actually use this technology.
Challenges we ran into
There were a lot so I'll bullet point
- Crazy Merge Conflicts
- Dependency Clashing
- The Tensorflow Graph is Broken ... Maybe it's actually me :/
- Slow Wifi
Accomplishments that we're proud of
We were able to bring it all together. There were a ton of moving parts in this model, from the preproccessing, to training deep predictive models, to sharpening video quality, to developing a slick UI and flask backend, we were just in a grind to get everything done and working up to standards.
What we learned
What's next for Aper.io
We are planning on running this on a much larger dataset that we can scrape from the web, try to get the algorithm to generalize much better, and publish a paper potentially detailing a rigorous evaluation of our various methods and outcomes so that other DL practitioners can build off our work, just as we built of others.