Project Neutronium - Beyond Compression

Reduce Image Size by up to 99.7% from PNG. Using machine learning services to crush image data down to impossibly small files sizes.

Comment

Inspiration

I've been learning how to draw and paint for over the better half of a decade, and while that's not my trade, it does inspire an idea: Why don't computers recreate images in the same way humans do? Human to human image recreation is surprisingly data efficient. Rather than relying on complex mathematics and the nuances of various matrix calculations, humans describe images based on an assumed common knowledge base that the recipient works to decode. A man tells a painter to paint a mountain at dusk, with clouds coming from the north as the sun sets behind the third peak. That right there was roughly 124 bytes, but most people have a fairly vivid mental image already. Why can't we have computers do the same?

What it does

Neutronium utilizes Google Vision API to understand the details of an image, learning the general setting and what to look for, before it goes through several stages of color quantization to break down an image into individual components. Next, the new regions of image are analyzed and categorized into various segmentation masks (sky, hill, mountains, etc.). Finally, the outputted neutronium can be run through NVIDIA's GauGAN to recreate the image.

How I built it

Neutronium was built strictly in React, with a Material UI frontend. It called the Google Vision API, while handling much of the additional processing locally.

Challenges I ran into

Google Vision's packages weren't meant for frontend use, the NodeJS support was intended to be run server-side. Accessing the API through a static web frontend and creating a powerful yet efficient algorithm to be run entirely client-side was the greatest challenge by far.

Accomplishments that I'm proud of

I'm rather impressed that it did get some things working. The whole concept of crushing an image down 99.6 percent and still retrieving recognizable data out the other end is rather fantastical, but it surprisingly worked for a number of images I've tested with.

What I learned

We have a lot to learn from how nature operates. Much of computer science is rigorously mathematical, and that misses out of the imperfect, wild solutions that spontaneously emerges in nature. That's the basis behind many machine learning algorithms, the origin of many a breakthrough in science, and what led me on my path to creating an incredibly imperfect way to send and store images.

What's next for Project Neutronium

There is a surprising amount of potential in this concept that I've yet to explore. First and foremost, I'd like to use machine learning to identify blobs of area in an image to more accurately mask it. I'd also like to use similar algorithms to correctly sort the blobs into the right color masks (water is particularly difficult to sort by color, as it often is reflective rather than a color of it's own). Next, I'd like to further reduce the file size of the neutronium, using vector graphics to further crush the amount of storage necessary. Finally, I'd like to take each and every part of Neutronium offline, so that the promise of extemely low bandwidth image transmission is possible.

Built With

Share this project:

Updates