The Nintendo Switch JoyCons are full of sensors and inputs. Being able to access them makes the JoyCons capable of anything.
The idea of being able to access the gyroscope and accelerometer could make for some funny projects. Along with that, pretending they're instruments adds to the fun!
What it does
The web page displays waves and sounds based on the input from the JoyCons. The left JoyCon acts as a theremin, whereas the right JoyCon acts as a snare and drum.
How we built it
As it turns out, getting JoyCons to work in the browser is actually not very easy (surprise, surprise). So, we hacked an existing driver, written in Go, to interpret motion controls and present them to the Web Gamepad API. In the browser we have a Web Audio API generating sounds and an HTML5 canvas displaying visualisations.
Challenges we ran into
- The GamePad API doesn't officially support any sort of motion control. Using an existing driver, Dan added support for motion control, also so it worked with the GamePad API
- Trying to make the sound smoother and fade in nicely
- Getting the JoyCons to connect and not randomly disconnect
Accomplishments that we're proud of
Managing to get any motion controls from the JoyCons and then using that to produce sounds. Also, making the sounds smoother and the responsive interface.
What we learned
- Dan learnt some Go and learned about bluetooth drivers
- Justin learnt more about the Audio API and project management for git. Also, to only work with one browser during a hackathon.
What's next for Notendo Tunes
Adding more sounds to allow the user to make more music!