Gesture Controlled Soundboard

• 

  Gesture Customizer

• 

  Thumbs Up

Inspiration

We wanted a way for us to play sounds without having to tab out of our current browser or application.

What it does

Use media-pipe's hand gesture recognition model to analyze video from webcam, determine hand gestures, and then play corresponding audio files. Allows users to sign up, log in, upload audio files, and customize their gestures to their own liking

How we built it

With Angular frontend (Capacitor for Mobile), Django backend, MongoDB to hold user information, and Google Cloud to store audio files

Challenges we ran into

Hosting frontend on vercel, working with hand_landmarks in media pipe to build custom gestures, Deploying on mobile,

Accomplishments that we're proud of

Deploying, working with mp3 files, dipping our toes into hardware

What we learned

working with mp3 files, working with video/images, integrating Google Cloud, Capacitor

What's next for Gesture Controlled Soundboard

Expanding on custom hand guestures, double hand gestures, combination gestures, implementing face landmarks to expand user control of the soundboard

Built With

• angular.js
• capacitor
• django
• google-cloud
• media-pipe
• mongodb
• python
• typescript