EduPiano - old ver.

We all struggle with learning the fingering of a new piece unless its already mentioned. Well, now that problem is solved.

Comment

Inspiration

I have a piano exam this Friday, August 23rd, and while learning my pieces, I struggled to find the most comfortable finger positions. I believe every pianist has wished for a tool that generates piano fingerings and guides them step by step (since we don't always have a piano teacher around). So, I took on the challenge to create one, aiming to make it as accurate as possible by considering different hand sizes. Please enjoy!

What it does

The project generates the "best" fingering for a piece based on the user's hand size and other parameters, annotating them directly onto the sheet music. It then uses a 2D keyboard in Pygame to show the fingering and hand positions. Finally, after setting up the projection light on your piano/keyboard, it teaches the user step by step what notes to play and which fingers to use. While it may not be 100% accurate, it provides a solid foundation for users to build on.

How I built it

I built the entire project using Python and its libraries. For reading and annotating the sheet music with fingerings, I used the music21 library and some fingering-generating algorithms. The sheet music is read and annotated by controlling the elements of the .XML file and adding new elements like fingerings (this is done in the "annotate.py" file). The fingering algorithms assess how suitable or comfortable a finger arrangement would be for a group of notes, considering variables like movement velocity (handled in the "Hand.py" file & class).

The 2D keyboard and sound elements are relatively simple but tedious. The keyboard is built using Pygame, with shapes representing the keys, hands, and fingers. For the sound, I manually recorded all the notes and applied waveform math to play notes and chords. All of this is in the "keyboard.py" file. Since I was already using Pygame, I decided to make a GUI on Pygame. Luckily I was able to do this quickly, but it came out basic, and not very appealing to the eyes. I wish I had more time to develop it better so that there would be more user freedom. Specifically, I wanted to allow the user to upload .XML files, .PIG files, and .MIDI files, but the only one that worked was .XMl and it was the first one I found. In hindsight, I should've built the GUI on a domain or website, because it would've been easier to upload and share the output files.

Finally, I built a projection piano/keyboard that displays light in the form of a piano (or even on an actual piano) and goes through the notes step by step. The flashlight shaper, shaped like a piano with two hands, projects the piano keys and hand positions. As the piece progresses, specific keys and fingers light up to teach the user how to play in real life. I wasn't able to finish this part, as I'm still deciding on how to show which keys are pressed and which fingers are used. My options are to have individual light sources for each key and finger or use controllable "flaps" in the flashlight shaper to highlight the keys and fingers. Instead of updating on Pygame, it would control the light sources or flaps to display the output.

Challenges I ran into

Fingering Generation

  • Creating a class for each note was tedious and time-consuming, especially since I started this project only a week ago and was under tight time constraints. -I also had to physically check fingerings on my piano, noting which were realistic (like avoiding the ring finger crossing over the middle finger). I even considered using machine learning, but found it too complex for the time available. In future versions, I plan to incorporate 3D simulation data and machine learning.
  • Annotating and reading the sheet music .XML files was challenging due to the difficulty of understanding the format and the scarcity of tutorials.

2D Keyboard Construction

  • While the keyboard itself wasn't hard to build, creating the hands and fingers was tricky. I'm not very experienced with Pygame, so when trying to rotate the fingers for realistic movement, I struggled and still don't fully understand why it didn't work.
  • I'm also not satisfied with the graphics, as they are just rectangles and the program is slow. The slowness comes from the sequential playing of notes and the need to maintain the correct duration for each note. (I sometimes wish I could've used C++ or Java for better performance.)
  • Running and listening to the program was also frustrating, as I had to re-run the annotating file every time (since they shared uncopiable information), which took about 30 seconds depending on the piece's length. This isn't a major problem but becomes annoying with constant repetition, especially under time pressure.

Music/Sound Playing

  • I manually recorded every note and modified them using soundwave software. While I could've downloaded them online, I didn't have the budget and wanted the "full" experience (a decision I now regret).
  • The main challenge arose with chords. I tried various methods to construct chords: summing up recorded waveforms, using sinewaves to generate notes, and more. The first two methods were somewhat successful but not perfect, and I couldn't find good tutorials, making it very difficult.

Accomplishments that I'm proud of

Although I'm not fully satisfied with the 2D keyboard graphics and sound effects, I believe I created a relatively usable tool for pianists. I tested it on one of my new pieces, and about 83% of the note fingerings were correct (except for some crossovers). Overall, I'm proud of the decent accuracy achieved. With more time, I could refine the project and possibly publish it.

What I learned

As with every project, I learned more about different Python libraries and how to manipulate them to fit my needs. I also discovered that it's possible to generate sound using sinewaves, something I knew theoretically but hadn't tried to code before. I gained a deeper understanding of Pygame, which will help in future projects. I also realized that there are many solutions to a problem, and proper planning and research can significantly improve efficiency and decision-making.

What's Next for EduPo: Piano Fingering Generator & Teacher

One of my ideas for this project was to create a Piano Posture Fixer, but I ran out of time. Using existing posture computer vision Python programs, it would analyze the user's piano posture, suggest corrections, and explain the benefits. This feature would be extremely useful for beginners who tend to slouch, bend their fingers, or sit too comfortably. Correcting posture—like straightening your back, maintaining the right distance, and curving your fingers—can immediately improve control over the volume and articulation (staccato and legato) of the notes. In the future, I plan to expand this project to support multiple instruments, like the saxophone, trumpet, flute, percussion, strings, etc. I would apply the same fingering generation, 2D to 3D simulation, and music features to these instruments. To make it organized, I would develop it into a proper app or website with music theory resources and a platform to connect with other players and find future band members or teachers. This would transform the project from a one-week hackathon speedrun into a functional and notable app.

Built With

Share this project:

Updates