Durhack winning project

Inspiration

Initially our inspiration was to create a tool to generate music and art using multi-sensory inputs from the surrounding environment. Communities can collaborate with the goal of creating unique art that expresses their culture and what they stand for. We feel it is important that our tool is accessible to everyone. After viewing the ALGORAVE, we got heavily inspired to incorporate mosaic art and displays in our project to allow for users to better express themselves.

What it does

The project allows multiple users to join a virtual room and contribute to the creative expression by streaming the sensory data they collect from their surrounding. All the streams get sent to a backend server that collects and fuses them using our algorithm to birth something new and exciting that reacts to live changes in the input streams. For example, users can use their hand gestures to manipulate the pitch of the audio, can shine light at the light sensor to DO SOMETHING, or can move the ultrasonic sensor closer or further away from a surface to see how that the output changes.

How we built it

When phones connect to a room their camera and mic stream to a Python backend server over WebRTC. Then using Next.js we display a dashboard which shows each users camera feed in a grid.

Our pinch gesture tracking runs on the server-side. We are running MediaPipe using OpenCV to detect the users hand and the subsequent pinch gesture they make. Detected changes are sent back to the client to allow for accurate vocal modulation via Tone.js PitchShift. When the users' fingers are getting closer together the pitch of their voice is decreased and vice-versa when their fingers are getting further apart.

We constructed the hardware using an Arduino uno Q to take an input from multiple sensors. We used breadboard as a way to create a positive and negative rail in order to power the sensors, these were connected to the 3.3 v and ground pins on the Arduino. We used 2 digital pins one as an input and one as an output for the ultrasonic distance sensor having one of the pins set as an output to cause an ultrasonic burst with the other pin being input in order to read the duration of the return echo pulse and then calculating the distance from this. For the sound sensor we used an analog pin so we could use analogRead to get the sound level, for what we were doing there was no need to convert to decibels so we used the pure analog value. For the light sensor this required no wiring as it simply plugged into the Arduino, and we then pulled the ambient light in lux, the infrared intensity and rgb values from the sensor. We then sent the data to the server using rest APIs so it was able to be used to make changes to the website.

Challenges we ran into

We run into some difficulties detecting the pinch at first, and debugging was made more difficult by the fact that we were ignoring the root exception. The fix was wrapping frames in mp.Image and making sure no errors are silently caught and ignored.
The pitch effect took multiple attempts before getting the right shift amount down.
The laptop camera can only be held by one process, so every browser test meant killing the Python tracker first.
Security measures implemented in most modern browsers forced us to make all components in our pipeline expose an HTTPS endpoint.
And, of course, some standard git drama, with merge conflicts so intricate they make you wonder why people don't just go back to naming the folder "Version2Final" instead.

Accomplishments that we're proud of

Joining a room is as easy as opening a web page on your phone's browser.
We efficiently handle the computer vision on the server and perform all additional computation in one detection pass.
Your hands are your instrument.
We reveal the music and art hidden in your surroundings.

What we learned

Planning and establishing an MVP is the key to not lose track of the goal.
Silencing exception is never a good idea.
Branching and pull requests are the only sane way to collaborate when multiple people are working on the same repository at breakneck speed.