Keylusion

• 

  Login Screen

• 

  Piano page when hand detection & camera is on

• 

  UI of Piano Page while camera off.

Inspiration

We wanted to create a computer vision piano key web application which you can play piano with your finger tips when you curl your fingers down. No physical piano required!

What it does

Keylusion is a browser-based virtual piano that lets users play music using hand gestures. Using their webcam, users can see their hands being tracked in real time. Each finger is assigned a musical note, and curling that finger triggers its sound. The note continues playing while the finger stays curled and stops when the finger straightens.

The website also includes a clickable virtual piano, visual feedback showing which notes are being played, and user authentication. We also implemented a Loop Station that records note sequences and allows users to replay melodies while continuing to play.

How we built it

We built Keylusion using React, JavaScript, HTML, CSS, and Vite for the frontend.

For the computer vision component, we used Google's MediaPipe Hand Landmarker to detect and track hand landmarks through the user's webcam. We developed custom finger-curl detection using joint angles and fingertip distances to determine when a note should start or stop.

We integrated Tone.js to generate musical sounds in the browser and handle multiple notes simultaneously. We also used Supabase for authentication, allowing users to create accounts and sign in.

Challenges we ran into

One of our biggest challenges was making hand tracking feel natural and responsive. At first, detecting finger movement consistently was difficult because small changes in hand position could accidentally trigger notes. We worked on calibrating hand positions, smoothing landmark measurements, and distinguishing between curled and extended fingers.

Accomplishments that we're proud of

We're especially proud of combining real-time hand tracking with interactive music generation in a single website. This include tracking both hands through a webcam using MediaPipe, mapping individual finger gestures to musical notes and building a custom finger-curl detection system.

What we learned

This project gave us hands-on experience with computer vision, audio programming, frontend development, and collaborative software engineering. We learned how MediaPipe processes hand landmarks, how to translate those measurements into meaningful gestures, and how Tone.js can generate and control sounds in real time.

What's next for Keylusion

Built With

• computervision
• css
• html
• javascript
• mediapipe
• react
• react-native
• supabase
• vite