About the project
Inspiration
Learning piano often starts with reading sheet music and following visual demonstrations. We wanted to explore a different starting point: learning through listening, playing, and conversation.
We built ADA Piano Teacher for learners who cannot rely on visual sheet-music interfaces, with blind and visually impaired learners as our intended audience. Our goal is to make practice guidance easier to hear, understand, and act on.
What it does
ADA combines a MIDI-connected piano, spoken interaction, score import, and camera-based hand tracking in one desktop prototype.
Learners can ask piano questions aloud and hear ADA’s responses. For guided practice, they can import MusicXML or convert a PDF score, review and approve a passage, listen to a reference performance, and practice its notes and chords.
ADA uses MIDI events to check what the learner actually played. The AI coach receives those measured results and suggests a small next practice step. Camera hand tracking is also available, though it does not yet assess fingering or technique.
How we built it
We built the application in Python with a Tkinter desktop interface. Mido and python-rtmidi connect the application to a digital piano, while a local evaluator checks note and chord accuracy.
Audiveris converts PDF sheet music into MusicXML. Imported passages require review and approval before they become practice material.
The AI coach uses OpenAI’s Responses API. Linux voice input uses recorded audio and OpenAI transcription, while text-to-speech produces spoken replies. OpenCV and MediaPipe provide camera capture and hand-landmark tracking.
We use uv to manage Python dependencies and GitHub Actions to run automated checks.
Challenges we ran into
One challenge was keeping musical feedback grounded in evidence. We separated performance evaluation from AI coaching: MIDI determines which keys were played, and the model explains the results.
Another challenge was coordinating microphone input, spoken replies, reference playback, and the interface without blocking interaction or letting ADA hear itself. We added background processing, cancellation, and microphone pauses during spoken output.
Score recognition introduced another source of uncertainty. Instead of treating every converted PDF as correct, we built a review-and-approval step before practice begins.
Accomplishments that we’re proud of
We connected score import, audible reference playback, MIDI-based practice, and conversational coaching in a working prototype.
We also implemented Linux voice interaction and verified microphone capture, cloud transcription, spoken playback, and live agent responses. Our local verification included 217 passing automated tests.
What we learned
An accessible learning experience needs more than spoken interface text. Instructions must be concise, controls must remain predictable, and feedback must distinguish measured facts from assumptions.
We also learned the value of keeping local practice independent from cloud services. Learners can continue basic note and chord practice even when AI coaching is unavailable.
What’s next for ADA Piano Teacher
Our next priority is testing the complete workflow with blind and visually impaired learners, especially score review, keyboard navigation, and screen-reader compatibility.
We also want to improve voice interaction, add rhythm assessment, and investigate calibrated finger-to-key tracking. The current prototype evaluates note and chord attacks; it does not yet grade rhythm, posture, or fingering.
Log in or sign up for Devpost to join the conversation.