Synaesthetic

Inspiration

Synesthesia is a phenomenon where one of a person's senses being stimulated causes an unrelated sense or experience to occur. The term "synesthesia" comes from the Ancient Greek words 'syn', meaning "together", and 'aisthēsis', meaning "sensation" - translating literally as "combined experiences".

Generally synesthesia can be classified into two forms. The first is associative synesthesia, where a person feels a very strong and involuntary connection between a stimulus and an unrelated concept or experience. One of the most common examples of this is grapheme-color synesthesia, where letters or numbers are strongly associated with colors. Interestingly, people experiencing synesthesia tend to have the same or very similar associations. In this case, if you ask ten people with grapheme-color synesthesia about a list of numbers, they are likely not only to associate each number with a color but also to choose the same colors as others with the condition.

The second form is projective synesthesia, where a person encounters a stimulus and an unrelated sense is triggered in response. For example, chromesthesia is the association of sound to color. A person experiencing associative synesthesia may hear a particular sound and think strongly that it "sounds blue". However, a person experiencing projective synesthesia may hear the same sound and "see" a blue color in their mind's eye, as if they were looking at a blue piece of paper. In fact, these projection experiences tend to have extremely similar brain neuron activity to the physical sense triggering.

Synesthesia is often a very difficult concept to comprehend. While grapheme-color synesthesia is one of the most common forms of synesthesia, it is estimated that only around one percent of the population is capable of experiencing it. Projective synesthesia tends to be particularly difficult because it is significantly rarer and there are few experiences quite like it.

What it does

Synaesthetic (a portmanteau of "synesthesia" and "aesthetic") is a program designed to bridge the gap between synesthetes and the general population. A user provides the program with an audio file and the software procedurally generates a visual representation of the audio similar to what a person with projective chromesthesia would experience. Not only does this allow everyone the opportunity to witness the phenomenon themselves, but it also provides a tool for education on an experience that is truly unlike any other.

How it was built

The program runs entirely on an Apache web server inside of a specially designed Docker container. The Apache server is hosted on an Amazon Web Services (AWS) EC2 instance, which is configured to allow internet access over HTTP using a domain registered through GoDaddy.

The driving force behind the software is JavaScript. Using Audio() objects, a user-provided audio file is taken, assigned an audio handler, and parsed into core components that most strongly associate with chromesthesia, such as the frequency and amplitude of the sound. This is done using the Fast Fourier Transform (FFT) algorithm, an acoustics measurement method used to convert audio signal into individual spectral components. Once these components are successfully parsed, the program then uses custom-built algorithms to associate the sound with a color that approximates the state of the sound as a synesthete would experience it.

Once the color(s) are calculated for the audio at a given point in time, the remainder of the script handles the visual element. A JavaScript canvas is used as the baseboard for Particle() objects to be drawn on. While some people with chromesthesia experience distinct shapes as well as colors, most often the colors are shapeless, and the Particle() objects are a very close approximation of the aura-like effect that the colors present themselves as. The canvas is constantly being updated as the audio progresses, providing a fluid and seamless experience throughout.

While the infrastructure posed some challenges in setting up, most of the time put into this project was into the minutia, optimizing and tweaking the small values to create the most realistic experience possible, from the shades and hues of the colors to the aura effect to the frequency of change in the colors. Normally this would be a very difficult process to describe in order to get the details just right, but it certainly helps when the lead developer of the project just so happens to have chromesthesia of their own. :)

What I learned

In high school, I took a very basic course on web design that covered only HTML and the basics of CSS. I did little JavaScript work at that time, and what I used was essentially copied and pasted from other sources online. I even competed in state-wide web design competitions knowing absolutely nothing about JavaScript. I knew going into this project that I wanted to challenge myself to finally learn at least a bit about the language. What I didn't know is that most of the program's functionality would have to be done using JavaScript, which provided a unique challenge as I didn't know if my vision for the project was feasible for the short timeframe or even possible at all. However, after hours of reading through documentation and watching beginner tutorial videos, I'm extremely proud at the end product and how professional it ended up looking.

Outside of JS, I also had to do a good amount of research on acoustic processing and how exactly audio signals get converted into digital signals that a computer can interpret. This research was one of the biggest breakthroughs for the project, as understanding how the audio components worked under the covers made it much easier to use JavaScript audio objects and overall gave a lot more context into how sound relates to visuals through synesthesia.

Applications

The core application is to provide a tool for the general population to be educated on the marvel that is synesthesia. Moreover, it is a platform that aligns harmoniously with the mission embodied Panera: to create a healthier and happier world by blending innovation with compassion. It allows all people to empathize with synesthetes and better understand that we don't all perceive the world in the same way.

In the same vein, Synaesthetic also can be used as a tool for the hearing impaired to have a new approach to sound that may have otherwise been impossible. While modern medicine can help to strengthen hearing ability in some cases, it may be too expensive to receive the care required. In other cases, the hearing loss may be too severe to be repaired at all. Synaesthetic is a free program that works with any audio, whether it's a small sound bite or a full orchestral performance recording.

Another application is in the music industry. It is very common for a director of a group of musicians to look at a music score and have a particular sound in mind. As such, they try to use as many descriptive words as possible in order to convey to the other musicians how the music should sound. Terms like "airy", "weighted", or "moody" are commonplace in these scenarios, but often this gives the musicians only a vague idea of what the sound in question is. Synaesthetic provides a new approach: play the music in question and use the program to visually represent the sound, creating a much less ambiguous picture of the music. In fact, this concept is not unfamiliar - famous romantic composer Franz Liszt is known to have described music with color to his groups, using phrases like "play it more blue". (This also happens to be one of the earliest recorded instances of a potential chromesthesia experience.)

What's next for Synaesthetic

If Synaesthetic could be used in the music industry to review sounds in a visual manner, then it is also plausible that the reverse can be a useful tool as well. Providing support for turning colors into abstract sounds has the potential to open up a whole new approach to composing music, as instead of following the strict music theory rules, the composition can instead be described by the experience it is meant to invoke in the listener.