The product, synesthesEAR, is targeted at the inner artist in anyone. The psychological condition, synesthesia, is diagnosed in people who associate certain sensory inputs to another type of sense (for example, seeing a red stop sign makes them taste a particular food or hear a particular sound). We aim to do just that, reading colours as sounds.
This will be done with the use of an RGB sensor attached to the end of a sort of “magical wand”. The RGB sensor will convert the frequency of light emitted from the artwork into 3 values ranging from 0-255. Each value corresponds to either Red, Green, or Blue. For our first deadline on the 3rd of April, each of these colours will be correspond to a tone, high pitch, medium pitch, and low pitch respectively from three buzzers. The value of each colour on the RGB scale determines how loud the respective buzzer plays. For the more complicated version due on the 17th of April, we plan to use MatLab to incorporate a speaker. We will match the frequency of the sound to the RGB spectrum. This way, it’ll produce a higher quality sound.
Our product, the SynesthesEAR, was born when we learned in Psychology class that some people saw certain sounds as colors, colors as taste, or sound as tastes. We were intrigued by this concept of Synesthesia, a condition where the brain's area for the sense overlap, allowing different stimuli to be perceived by different senses. Thus, we wanted to create a device which allows one to translate hues into sounds.
By the first demo day, we had understood how our RGB sensor works and the code which is needed for it to work. We attached the sensor to an RGB LED in order to demonstrate to the TA's that we are able to detect the color of an object and transmit it through the LED. Although the RGB LED was a little unreliable as the wire was poorly connected to it, it was clear that the RGB sensor was registering what it should from viewing the serial monitor. By the final demo day we made sure to take out the lights and put in a speaker/buzzer instead. This also requires us to convert the output. At first we simply fed the values of Red, Green, and Blue to the RGB LED; however, using a speaker, we had to convert the values of RGB into a frequency. This required a lot of careful consideration because, at first, we planned each color to reflect a different frequency, but the problem we found is that there was no linear color scale to match to a linear frequency scale. As a result, we decided to create two different programs (modes) to run on our device. The first mode allowed the device to read the most prominent color, red, green, or blue, and the second mode allowed the device to read the shade of the color.
First, let's look at the pen. The pen is simply connected to a Bluetooth master module and an RGB sensor. Only the code is different for each mode. For the first mode, the different values returned from the RGB sensor for each color is compared and the highest color value is retrieved. If red, the Bluetooth module sends a "R", and "G", and "B" for the other two colors. For the second mode, the Arduino goes a step further and takes the most prominent color and depending on how intense it is, returns different values such as "A", "B", "C" and so on.
Secondly, looking at the speaker component. The hardware is fairly simple; the Arduino is hooked up to a buzzer and a Bluetooth Slave module. For the first mode, the Arduino simply plays a different tone based on whether it received an "R", "G", or "B". For the second mode, the Arduino also plays a different tone based on whether it receives an "A", "B", "C"… "H". Most of the difficult coding lies in the pen component.
This project, once made more sophisticated, could be marketed as a pioneer of a new form of artwork, bridging the gap between sight and sound. Psychologists theorized that those with synesthesia tend to be more creative; this device will allow everyone to experience what those with synesthesia experience.
To improve on the product, we will firstly create a nice housing for the speaker and the Arduino in each subsystem. This will be either done with 3D-printing or laser-cutting. The wire connecting the Arduino to the pen will also be extended to allow for easier mobility. As for the software itself, we hope to extend beyond simply being able to translate shade to tones. We hope to be able to map the non-linear RGB color palette to a tone from high to low.