Lyra - Conversational AI Piano Teacher

Lyra listens and understands piano input
Practice chords, scales or ask questions about what you're playing
Lyra is the first conversational AI piano teacher

Inspiration

Most piano tools treat music like static content: fixed lessons and videos that never listen back. As the team behind Museverse, a VR music school, we’ve heard many versions of the same request: “I just want to ask a quick question and have a teacher show me on the piano.” Lyra is our experiment in what happens when an AI can hear and play music, not just talk about it.

What it does

Lyra is a voice-first piano companion that runs in the browser. You connect a USB-MIDI keyboard and talk to Lyra as if she were a teacher next to you. You can ask things like “Teach me a C major chord”, “What chord am I playing?” or “Can we practise a G major scale?”. Lyra plays examples, listens to you, then answers in natural speech with encouragement and precise feedback – for example, telling you that you played C minor instead of C major, or that you missed a note in the scale. The conversation itself becomes the lesson.

How we built it

Lyra runs on ElevenLabs Agents with Gemini 2.5 Flash as the reasoning model. We define tools that let the agent play notes, chords and scales, and send structured MIDI events to the browser. A web client using a modern JavaScript stack and the Web MIDI API talks to any connected keyboard, visualises pressed keys and routes messages between the instrument and the agent in real time. The whole experience is hosted on Google Cloud Run, with Gemini used to keep voice, reasoning and MIDI feedback low-latency and scalable.

Challenges we ran into

The hardest part was making the interaction feel like a relaxed, human lesson instead of a noisy stream of notes and words. Early prototypes tried to play and speak at the same time, which overwhelmed users, so we choreographed when Lyra talks, plays and listens. We also hit the current limits of LLMs for music theory comprehension so we focused v1 on chords and scales where Gemini performs more reliably.

Accomplishments & what we learned

In a few weeks we went from idea to a hosted app where you can ask for a chord, play it on a real piano and get instant, voiced feedback. We learned that conversational, voice-first UX is a natural fit for instruments, because your hands stay on the keys, and that ElevenLabs Agents plus Gemini already support rich musical dialogue when paired with good tool design.

What's next for Lyra and Museverse

Lyra is the first step towards our larger vision of personalised AI music tutors across web and XR. Next, we plan to add memory so Lyra can track progress across sessions, expand from chords and scales to short pieces, and connect this browser-based teacher to Museverse’s VR piano lessons on Quest. Over time, we imagine Lyra living wherever people practise – laptops, headsets and future wearables – turning any piano into a smart, conversational instrument.