Wuqiong had always been interested in the interaction of art and science. He had the idea of mapping words to fractal animations to music to dance moves to create a comprehensive form of art that could be experienced through all the senses and cross the boundaries of all art media. Eric is interested in game design and graphics. Nat is interested in web development. Kevin is interested in backend development. We are all good friends with a variety of skill sets and interests that we could combine together to create an interdisciplinary project the would has never seen before. We were able to combine our different perspectives to brainstorm unique ideas and implement them.

Before the hackathon, Eric prepared Leap Motion and Raspberry Pi hardware so that we will be ready to implement any ideas we might have. Everyone discussed the ideas together and realized that we can use Leap Motion as the inputs to generate the fractal animation and music. We implemented the basic fractal growing and shrinking function first, then we added the functions of changing the colors and saturation of the fractals through different movements. Next, we added the ability to produce sounds through HTML5 Web Audio. We were inspired by theremins and their ability to create oscillating sounds, magically. Nat used the Web Audio's oscillator allowed us to manipulate a frequency, and he used the inputs through Leap Motion to change the inputs. We were able to change the frequency with a roll gesture, and volume with pinching, as well as add drum sounds with sudden changes in the y-axis. Eric and Kevin started out working on the fractal patten. We were interested in the Mandelbrot set, but iterating over the pixels to redraw frames is hard to do smoothly in real time. We found Andrew Hoyer's implementation of the H-tree fractal very appealing, and it changed with mouse movement smoothly with javascript.

One of our first barriers was understanding using Leap.js's way to get input from the Leap Motion device. After we realized that it is in a very simple loop, that returns json objects with states of hands, we very quickly progressed to get it to start giving the fractal and the oscillator input.

Our goal originally was to create a user experience that involves many senses, and to be engaging and fun to use. The fractal tree and the oscillator share some common gestures, so the user can get both visual and audio pleasure from moving themselves. People who came by and saw us while we were developing this hack had a lot of fun while playing with our project, and some even achieved Treegasm.

Built With

Share this project: