Harmonic

Harmonica is a software that allows users to learn instruments by air playing them, avoiding the expensive costs of learning an instrument. It allows users to learn, record, and play with friends.

Comment

Inspiration

Kids regonize sound and color long before they can read, but most learning apps still throw text at elementary schoolers. Inspired by the STEAM Initiative, we decided to focus on the Arts part, a distinctive experience offered by Milpitas Community Education Foundation. Music tutors, or expensive instruments can act as a barrier to entry for many kids exploring and learning different instruments. We have decided to close that gap, and bring music and learning to anyone with a computer and a camera.

What it does

Harmonic is a 3D music room in your very own browser. You pick an instrument, and you can move into two modes, Composer or Music Learning. In Composer, the child can express their creativity by playing different instruments however they want. In Music Learning, they can upload sheet music, or select one of the pre-selected songs loaded into our data base to learn and play them. Making it extremely easy and convienient for everyone.

How we built it

We use Three.js, WebGl, and GSAp for camera moves, bundled with Vite and every instrument is generated by code instead of downloaded. We coded our computer vision model, where MediaPipe reads the webcam and we map the hand landmarks into the Web Audio API to make sound, with a small state machine keeping the camera, the animation, and the audio in sync.

Challenges we ran into

Hand-tracking is usually pretty jittery, so sudden movements can cause multiple notes even when the user doesn't intend to do that. So, we smoothed the input and added a deadzone, and we had to cut the lag so playing felt instant.

Accomplishments that we're proud of

We got real-time hand tracking and sound running entirely in the browser with nothing to install, and built the whole 3D room and every instrument with code instead of downloading stock images and placing them there. The part we're happiest about is that a kid can step in and play anything that they want within seconds due to the feasibility and intuitiveness of our web design.

What we learned

Getting the input to technically work and getting it to feel like playing real music are two completely seperate and tedious tasks. Most of our time went into the second part, fixing the natural hand jitter, cutting the lag, building an intutitive design, and keeping a consistent visual style. All of this helps make the user experience feel special and unlike anything they have used before.

What's next for Harmonic

More instruments, plus a guided mode where kids follow along to play simple songs which unlocks rewards. We also want to add a studio mode where the child can full express their creativity by playing different instruments in a studio and also recording and downloading the audio/video. This can be a enjoyable experience. We also want to add a multiplayer mode that can spark collaboration between children.

Built With

Share this project:

Updates