Deep Correct: DL Color Correction for Color Blindness

Color Correction Results

Title

Deep Correct: Deep Learning Color Correction for Color Blindness

Who

Naveen Sharma (nsharm25), Taishi Nishizawa (tnishiza), Ian Rider (irider)

Introduction

Color vision deﬁciency affects 8% men and one in every 200 women. Our project member, Naveen, is one of them. He has difficulties distinguishing between red and green colored objects. This prompted us to ponder whether deep learning could be used to help with color blindness. After some research, we came across one research paper that uses deep learning to create a color correction program that can aid with object recognition.

We are implementing the following research paper: link by Petrovic and Fujita.

Their abstract: Abstract. Color vision deﬁciency affects 8% men and one in every 200 women. There are many different types of color blindness with the red-green as the most common. Most models for color correction are based on physiological models of how people with color vision deﬁciency perceive the world, with the goal of reduc- ing errors derived from the color blindness simulation formula. In this paper we present Deep Correct, a novel Deep Learning based method for color correcting images in order to improve accessibility for people with color vision deﬁciency. The key elements of this work with regard to color blindness are two-fold: 1) we propose a data-driven Deep Learning approach for color correction and 2) we cre- ate an objective, quantitative metric for determining the distinguishability of im- ages. Additionally, as a more general Deep Learning contribution, we propose a new method of training neural networks by utilizing error gradients from pretrained networks in order to train new, smaller networks.

This is an unsupervised learning problem, as the model does not use labels to train. It instead uses an architecture similar to that of a GAN.

Related Work

The Research Paper

[Very Deep Convolutional Networks for Large-Scale Image Recognition] https://arxiv.org/abs/1409.1556

The paper linked above describes a large convolution neural network trained on the ImageNet dataset. The paper describes the model of being able to “generalise well to a wide range of tasks and datasets.” The DeepCorrect model’s Corrector module is derived from the VGG16 architecture from this paper.

[Generative Adversarial Networks] https://arxiv.org/abs/1406.2661 This paper linked above introduces the GAN model. The DeepCorrect model we will be implement was heavily inspired by the GAN architecture, using a two component adversarial system similar to the G-D system described in the paper.

Data

This paper uses the Pascal dataset to train the different components of the DeepCorrect model. This dataset is publicly available online. As Pascal is a large dataset, only 11257 images were used to train the model.

Methodology

The DeepCorrect model is a GAN-like deep learning model trained on the Pascal dataset. It is composed of two main components: the Corrector and the Referee. The Corrector takes in image data and applies convolutional layers and a linear corrector layer to produce color corrected images. Then The Referee, which is similar to the Discriminator in GANs, determines the quality of the generated image. We believe the hardest part about implementing DeepCorrect will be creating the Referee network. As detailed in the original paper, the Referee is derived from the VGG16 model. It may be a challenge to fit the modified VGG16 into the rest of the DeepCorrect architecture.

Metrics

In the paper, the main goal was to create a model which generated superior color-corrected images than achieved using a linear color corrector. To verify whether or not the generated images from the model are better color corrected than the linear model, a color-blindness simulator is used. This way, numeric accuracies could be determined. Our main metric for success in this project will be to replicate the results achieved in the original paper within a margin of 5%.

Ethics

What broader societal issues are relevant to your chosen problem space?

Color blindness affects 8% of males and one in every 200 women, including one of our project members, Naveen Sharma. While there are various types of color blindness, the red-green type is the most common, with 98% of the cases. Color plays a big role in our ability to perform many daily tasks. The problem is especially emphasized in computer usage, where color is often a primary source of information distinction. However, it is not uncommon that color blindness correction comes as an afterthought to most developers. This is the most common approach for providing support for color blindness accessibility is image color correction. By modifying the software output image to contain colors which make desired elements clearly distinguishable, this serves as an integral part in helping color blind people navigate.

Why is Deep Learning a good approach to this problem?

To achieve image color correction, the most common solution is to apply a linear function that transforms the original color space into the new color space which is more suitable for people with a particular type of color blindness. However, this approach is limited. The primary purpose of color correction is to provide clear identification and differentiation of objects on the screen, and models that solely utilize the original images lack this crucial software-specific information. Using deep learning, we can train a neural network to output images that are most correctly distinguishable to color blind people. We simulate color blindness with software and evaluate whether the simulated results are distinguishable enough. This approach proposes a quantifiable method for estimating image distinguishability for normal and color blind people, giving us a clear metric to optimize and use for estimating correction method quality, and present a data driven approach for creating the color correction filter.

Division of labor

Our tentative plan is to work on each part of the project together, as we suspect each aspect of the project will require a great amount of effort. Since we are roommates, we plan on mainly working together in our room and continuously updating each other with how things are going.