Inspiration
Most people believe they cannot paint and cannot play an instrument. Almost everyone can wave their arms.
We wanted to give anyone 30 seconds in which they could be as unique as they'd like: move however you like, and leave with a painting and a piece of music that nobody else could have made.
What it does
It creates a unique song for every person based on how they paint. Then, using binaural audio (spatial) audio, during the demonstration, you get to hear specific sounds based on how you move your hands. After that, Gemini will analyse your painting, and ElevenLabs will narrate in the voice of an art historian, how it has interpreted this design. So imagine, just based on your movements, you get to create a special art piece and song customised to you
How we built it
Swift for main app Mediapipe for pose tracking Camera source -> Panasonic GH5 Apple packages for binaural audio
Challenges we ran into
Camera has incompatible software with my laptop. Too old, vs too new operating system. Solved via USB cable to laptop, then using iPad as second monitor to stream that connection, and view of the camera then use OBS to screen record the iPad screen, add it as a virtual camera, which allows Mac to input it as webcam footage to my program.
Accomplishments that we're proud of
The live trackin