AI Air Piano Coach

With AI Air Piano Coach, you don't need an actual piano to start with the basic lessons. With AI Air Piano Coach, you can do that just with a regular webcam on any flat surface.

Comment

Inspiration

Many aspiring musicians are deterred by the upfront cost and logistical commitment of purchasing or renting a physical piano. By leveraging modern web frameworks and commodity hardware like a standard webcam, we wanted to bridge this gap. The AI Piano Coach provides prospective musicians with an immersive first experience, allowing them to explore their musical interest and gain fundamental skills before committing to a significant financial investment.

What it does

The AI Piano Coach is a real-time, AI-driven Augmented Reality (AR) application designed to teach piano basics using common hardware. It transforms any flat surface into a virtual instrument, bridging the gap between digital tutorials and physical play through a sophisticated spatial logic engine and Gemini 2.0 Flash intelligence.

How we built it

The application is built as a Modular Monolith on the frontend, utilizing MediaPipe for hand tracking and Tone.js for high-fidelity audio synthesis. The backend is a lightweight FastAPI Proxy deployed on Google Cloud Run, facilitating secure, low-latency communication with the Gemini 2.0 Flash API. We utilized a hybrid inference model, running skeletal tracking on the client and complex depth analysis on the server.

Challenges we ran into

95% of our technical challenges centered on reliable key-press detection. We implemented and benchmarked 12 distinct algorithms—ranging from SVD decomposition and LSTM sequence analysis to ML Transformers—before arriving at our proprietary 'Differential Z-Anchor' method. This final approach significantly outperformed every existing open-source demo and software we tested, offering the stability required for actual musical practice.

Accomplishments that we're proud of

We are incredibly proud of achieving high-accuracy key detection on standard hardware without specialized depth sensors. Seeing the Gemini AI successfully instruct a student on hand posture in real-time was a "lightbulb" moment for us, proving that AI-driven pedagogy can be as intuitive as a human tutor.

What we learned

We learned that AR depth perception on flat, featureless surfaces is a profound challenge that requires multi-signal fusion (Y-velocity + Z-relative depth + Posture analysis). We also discovered that the "Conversational Agent" paradigm is far more effective for music education than traditional visual tutorials, as it allows the student to keep their eyes on the "keys."

Technical Appendix: The Path to Detection (12 Methods Tested)

Method Description Outcome
Differential Z-Anchor (Winner) Z-depth relative to wrist + extension posture (>155°). 95%+ Accuracy; stable across different camera angles.
Hybrid Kinetic Y-velocity + Z-spike + Finger curl angle fusion. Good for hovers, but prone to false triggers during surface contact.
Dynamic Floor Local minima tracking (Lowest recorded Y-point). Failed to distinguish between white and black key heights.
Motion Stop Halt detection (dy dropping to zero inside a key). Prone to noise from sympathetic finger movements.
Calibrated Y Velocity reversal after crossing a fixed Y-threshold. Highly sensitive to lighting and table surface reflections.
AI v1 (3D Plane) Least-squares fitting of a 3D plane to 7 landmarks. Very light-dependent and lacked the required precision.
AI v2 (Smoothed) Exponential smoothing (α=0.3) on Y-coordinates. Reduced "vibration" but introduced unacceptable latency.
PressureVision Hysteresis score based on Z-compression. Required specific sensor hardware not available to consumers.
MLP Classifier TensorFlow.js MLP trained on-device. High divergence; struggled with different hand sizes.
TypeNet LSTM Sequence analysis of fingertip movement patterns. Requires massive datasets of diverse players to generalize.
3D SVD Plane Robust surface fitting via SVD decomposition. Mathematically sound but computationally expensive for 60FPS.
Depth Anything Backend GPU Proxy for high-fidelity depth maps. High fidelity but introduced lag that broke the musical rhythm.

All the implementations needed to be tuned to ignore the fingers in certain cases for instance (understand some fingers were moving without having the "intention" to press a key)

Accomplishments that we're proud of / What we learned

Key press detection has a lot of improvement opportunities but considering it works in regular hardware with a regular webcam I am very happy with the result.

IA applied to images/video has a lot of potential. AI instructing the student to pose better with their hands is priceless.

Technical Specification: AI Piano Coach

1. Frontend Stack

  • Engine: Vanilla JavaScript (ES6+ Modules)
  • Computer Vision: MediaPipe Hands (21-landmark skeletal tracking)
  • AR Rendering: HTML5 Canvas 2D (Optimized for 60FPS performance)
  • Audio Synthesizer: Tone.js (Using Salamander Grand Piano multi-samples)
  • Voice Intelligence: Web Speech API (Speech-to-Text & Text-to-Speech)
  • State Management: Reactive Observer Pattern (Centralized in [state.js]

2. Backend Stack

  • Framework: FastAPI (Python 3.10+)
  • AI Engine: Gemini 2.0 Flash (via Google AI Studio / Vertex AI)
  • Deployment: Google Cloud Run (Containerized via Docker)
  • Networking: HTTP/REST for chat + WebSockets for real-time depth streams

🧠 AI Intelligence Layer (Gemini)

Gemini 2.0 Flash acts as a real-time "Musical Brain" for the application.

Multi-Modal Interaction

  1. Voice Coaching: Converts user queries into musical advice.
  2. Dynamic Exercise Generation: Gemini outputs structured JSON blocks that the frontend parses to instantly update the piano highlights. json { "exercise_name": "C Major Scale", "notes": ["C4", "D4", "E4", "F4", "G4", "A4", "B4", "C5"] }
  3. Real-Time Feedback: Plays are buffered and sent to Gemini for post-exercise analysis (e.g., "You missed the F#4, try keeping your hand arched").

📐 Spatial Logic Engine (Method 4)

DetectionMethod4 is the production-grade algorithm used to detect virtual key presses.

The Algorithm: Differential Z-Depth

Instead of absolute screen coordinates, Method 4 uses Skeletal Anchor Points:

  • Baseline: The Wrist (Landmark 0) acts as the depth anchor.
  • Signal 1 (Depth): Detects a relative Z-offset change > 0.04 between the fingertip and the palm.
  • Signal 2 (V-Angle): Calculates the joint angle (MCP-PIP-Tip). A press is only valid if the finger is extended (>155°), filtering out accidental hovers.
  • Signal 3 (Bounding Box): 2D Projection mapping for X/Y hit detection.

🎼 Sound & UX Design

  • Low Latency: Optimized AudioContext for <100ms trigger-to-sound latency.
  • Visual Feedback: Real-time "Key Travel" animation (4px downward shift) and glow effects for tactile-less feedback.
  • Responsive UI: Glassmorphism design system using CSS variables and Backdrop Filters for a premium, modern feel.

🚀 Performance Targets

Metric Target Implementation
CV Tracking 30-60 FPS MediaPipe WASM acceleration
Audio Latency < 50ms Tone.js Look-ahead optimization
AI Roundtrip < 1s Gemini 2.0 Flash + FastAPI Stream
Resolution 720p/1080p Adaptive canvas scaling

Built With

Share this project:

Updates

Submission history