Cleffy

• 

  Title Screen

• 

  Results Page

• 

  Music Sheet Scrolling

Inspiration

As musicians ourselves, there are two things that come to mind when asked what we hate about music: sight reading and practicing. Many of us wake up daily, subjugating ourselves to hours of brutal practice that drains us over time. Eventually, most musicians suffer from some form of burnout or aren't motivated enough to continue.

What it does

We created a prototype for a MMR based game where players compete in sightreading for ranks. Players select a game mode (competitive/casual/practice) and are presented with a piece of sheet music with limited time to analyze it. The player plays the piece, and the pitch/time accuracy is analyzed, displaying both overall and note-by-note accuracy in real time. By creating a multiplayer game where players compete on the basis of sight reading, we believe that we can radically change the way we think about practice; not only does it tackle a skill most musicians are bad at, but you can now have productive practice sessions with other people instead of practicing alone.

How we built it

Cleffy is a full-stack web application with a Flask and Python backend and a React-based frontend. We used PyAudio to process real-time microphone input and extract frequency data, allowing us to determine which note the player is playing at any given moment. The backend compares this input to the notes in a reference XML sheet music file and evaluates both pitch and timing accuracy. The frontend, built with React, handles the user interface, displaying the scrolling music, accuracy results, and match outcomes. This architecture is designed to be scalable, allowing for future expansion into mobile platforms and additional game modes.

Challenges we ran into

One major challenge was polling data rapidly and efficiently between the frontend and backend to maintain smooth real-time interaction. Another significant hurdle was fine-tuning the pitch recognition algorithm to accurately detect notes despite background noise, all while keeping processing delays minimal. This balancing act took up the majority of our project time.

Accomplishments that we're proud of

Developing a functional system that recognized the dominant pitch from an audio stream, and identifies its accuracy with a reference point. The algorithm is very efficient and works with <0.01s of processing time for a 0.1s audio sample section.

Built With

• css
• flask
• html
• javascript
• python
• react