Voice Piano

Voice Piano instantly turns your voice into a real-time piano accompanist, analyzing your pitch to generate beautiful, harmonically correct arrangements on the fly.

Comment 3

Inspiration

This project wasn't born from a technical problem; it was born from a human one. It started last weekend, inspired by a conversation with my wife. She has a deep love for music and is always humming new melodies, but like many people, she doesn't play an instrument. She turned to me and asked, "Wouldn't it be amazing if you could just sing an idea and hear what it sounds like with a full piano behind it, right away?"

That question stuck with me. How many beautiful musical ideas are lost every day because the person who imagined them lacked the years of training needed to express them on an instrument?

Voice Piano is my answer. It’s a tool designed to bridge the gap between imagination and creation. I wanted to build something that could have an immediate, tangible impact on people's lives—including my own. It’s about democratizing musical creation, making the joy of hearing your own melody harmonized accessible to everyone, instantly. It’s for my wife, for myself, and for anyone who has ever had a song in their heart but no instrument in their hands.

What it does

Voice Piano is a sophisticated web application that listens to a user's voice and provides intelligent, real-time piano accompaniment.

Key Features:

  • State-of-the-Art Pitch Detection A powerful, low-latency pitch detection engine using advanced algorithms (YIN, Harmonic Product Spectrum) in WebAssembly for near-perfect real-time accuracy.

  • Intelligent Accompaniment Not just note-for-note conversion. It detects the musical key and generates harmonically rich chord progressions.

  • Four Accompaniment Modes Switch between: Melody, Harmony, Bass, or Full arrangement instantly.

  • Musician-Centric Learning Tools Includes Scale Practice mode and a Chord Dictionary—making it both creative and educational.

How we built it

This project was built from the ground up over a single weekend, using cutting-edge web technologies for a native-like browser experience.

Architecture & Stack:

  • Platform & Deployment: Built in the Bolt.new environment and deployed globally using Netlify.

  • High-Performance Core (WebAssembly): Core pitch detection written in C++, compiled to WebAssembly for native-speed performance.

  • Frontend & Audio Engine: UI built with React, audio routing and synthesis managed using the Web Audio API.

  • Voice AI Integration: Integrated ElevenLabs Voice AI to allow natural language queries about music theory—turning the learning process into a conversation.

Challenges we ran into

The biggest challenge was achieving ultra-low latency in a browser. Initial JavaScript prototypes were too slow—notes lagged audibly.

Breakthrough:

We switched to a hybrid model: Offloading heavy signal processing to a C++ WebAssembly engine, enabling native performance with web accessibility. This was a pivotal architectural shift that made Voice Piano viable.

Accomplishments that we're proud of

  • Solving a Human Problem: This isn’t a tech demo—it’s a real musical instrument for real people.

  • Achieving Real-Time Performance on the Web: Our C++/WASM pipeline runs in any browser with minimal latency—pushing the limits of web audio.

  • Building an Educational Tool: Integrated music theory and practice tools for learners and hobbyists alike.

  • Rapid Execution: From idea to fully deployed app in a weekend—showcasing the power of modern dev tools.

What we learned

  • The Power of WASM: Practical insights into using WebAssembly for demanding audio workloads.

  • Modern Web APIs: Mastered the Web Audio API and integrated external AI services for a richer UX.

  • Hybrid Architecture Wins: Learned how to effectively combine React for UI and C++/WASM for core processing.

  • From Idea to Deployment: Used Bolt.new and Netlify for a seamless dev-to-prod pipeline.

What's next for Voice Piano

Upcoming Features:

  • Full Polyphonic Detection: Go beyond single notes—detect chords and complex harmonies sung by users.

  • ECKF Pitch Tracking: Integrate the Extended Complex Kalman Filter for pro-grade, sample-level analysis.

  • Collaboration & Sharing: Use Supabase or similar to enable users to save, share, and collaborate on their music.

  • Mobile First: Architect for native iOS and Android apps, maximizing performance on mobile.

Built With

  • ai
  • apis:
  • c++-frameworks/libraries:-react
  • cloud
  • elevenlabs
  • languages:-javascript
  • netlify
  • services/deployment:
  • typescript
  • voice
  • wasm)
  • web-audio-api-platforms:-bolt.new
  • webassembly
Share this project:

Updates