Neural Night Vision

Image 1 - Recreated Long Exposure
Image 2 - Short Exposure
Image 1 - Short Exposure
Image 2 - Recreated Long Exposure

Inspiration

We were inspired by the ingenuity of this paper and questioned whether is was possible to take this dataset that was built using sophisticated high-end cameras and applying it to the cameras in our pockets.

What it does

The entire dataset of Sony .ARW files were converted to jpeg images. The short-exposure, dark images are fed through in batch learning to a convolutional deep neural network with the long-exposure, bright images as the target data. We used the Adam optimizer and a loss function of mean squared error to evaluate pixel-by-pixel how close our network's guess was to the ground truth. Through this, the network, over dozens of epochs, was able to get better and better at brightening up a completely dark image.

How I built it

We used Keras with Tensorflow backend to build the convolutional neural network. Ali wrote the Java and Javascript code to build an Android app to get a camera stream, and John parsed and preprocessed the dataset.

Challenges I ran into

The main issue that we had to deal with was how to fit thousands of large RGB images in the memory. We made compromises like loading them in by batches of a given size, yet we still had to scale the images down heavily and convert to grayscale, even as we ran our dataset through beefy supercomputers on the Google Cloud Platform.

Accomplishments that I'm proud of

This is the first convolutional neural network that any of us have created, and we had to deal with a really small, imperfect dataset, which made training really, really difficult. We had to carefully inspect how every hyperparameter affects the training process to find the perfect balance between overfittment and underfittment.

What I learned

Practical aspects of building and training neural networks on small datasets of images, how to read research papers, and the many possible applications of machine learning.