XylophoneHero

Inspiration

A deep and unreasonable love of xylophones

What it does

An air xylophone right in your browser! Play such classic songs as twinkle twinkle little star, ba ba rainbow sheep and the alphabet song or come up with the next club banger in free play.

We also added an air guitar mode where you can play any classic 4 chord song such as Wonderwall

How we built it

We built a static website using React which utilised Posenet from TensorflowJS to track the users hand positions and translate these to specific xylophone keys.

We then extended this by creating Xylophone Hero, a fun game that lets you play your favourite tunes without requiring any physical instruments.

Challenges we ran into

Fine tuning the machine learning model to provide a good balance of speed and accuracy

Accomplishments that we're proud of

I can get 100% on Never Gonna Give You Up on XylophoneHero (I've practised since the video)

What we learned

We learnt about fine tuning neural nets to achieve maximum performance for real time rendering in the browser.

What's next for XylophoneHero

We would like to:

• Add further instruments including a guitar and drum set in both freeplay and hero modes
• Allow for dynamic tuning of Posenet based on individual hardware configurations
• Add new and exciting songs to Xylophone
• Add a multiplayer jam mode

Built With

• posenet
• react