LISA (Learning Images with Sound-based Assistance)

Inspiration 

Having always wondered how we can transfer sensory informations, especially for people who have limited senses (visually-impaired, deaf, etc.), we were inspired by a YouTube video featuring the Hilbert Curve, one which could branch throughout a plane and access inherently 2-dimensional data on a 1-dimensional scale. We joined this mathematical object with our passions to help impaired individuals by creating a method to convert between pictures and sounds in a meaningful, scalable, and understandable way.

What it does 

LISA is a program that is meant to convert visual data to audio data. Specifically, we map the HSB (hue, saturation, and brightness) values of each pixel in an image to an FTV value of a particular instant in the audio file. The order in which we choose the pixels matters a great deal, because we wish to choose them in such a way that any size image would be recognizable in the same shape as any other. The Hilbert Curve (a recursive shape) and its many pseudo-Hilbert Curves perfectly fit this requirement.

How we built it 

We used many applications and variations of JavaScript to combine some sound API's with Processing instances to make an all-around executable program.

Challenges we ran into 

Interacting with sound bytes proved to be very difficult for JavaScript to handle in a rapid way. As such, at a live speed, only a very powerful computer can output all sounds at the same time. However, preprocessing these audio-video demos can be done, and we have done this to show our intended product (proof of concept).

Accomplishments that we're proud of 

We were able to blend in three seemingly disjointed ideas: recursive math, image processing, and audio synthesis into one cohesive program.

What we learned

Technically, massive amounts of data cannot be handled in JavaScript when executing on a browser. Also, making the most generalized and versatile program takes a well-planned process to construct.

What's next for LISA (Learning Images with Sound-based Assistance) 

We hope to design a comprehensive training regimen to prove the viability of experiencing visual information through sound.

Built With

• css3
• html5
• javascript
• processing.js