Karacroakie: AI-Enhanced Karaoke Scorer

A fun, frog-themed AI-enhanced karaoke app that automatically generates karaoke tracks from song files, gathers lyrics, and rates the intonation of your singing compared to a reference.

Comment 1

Inspiration

As both musicians and STEM enthusiasts, we wanted to create a tool to help practice intonation and ear training. By comparing singing to original tracks, Karacroakie helps us and others improve. At the same time, it’s also a functional karaoke app with more traditional features, including lyric search and a leaderboard. This way, it can be used for both educational and entertainment purposes.

What it does

Karacroakie has countless features for smooth use. Once in the app, users can choose to either record themselves singing or upload an existing file of their singing If the user chooses to record, they'll first upload a wav file of the song they want to sing. Karacroakie automatically separates out the instrumental part from the vocals using AI. The user can search for lyrics using the title and artists or hit generate to transcribe lyrics from the audio. To record, the user can hit the record button and record themselves singing with the instrumentals. They can also play back their recording and choose to re-record. When the user is done recording, they press analyze to compare the two files. Meanwhile, if the user chooses to analyze an existing recording, they’ll upload the wav files for both the original song and their own singing. For each file, there are also playback buttons. Then the user can click the run button to compare the two files. Karacroakie compares the vocal parts of the original song and your recording and gives your singing a percentage score. After the resulting score is displayed, the user can choose to start over or save their results, which brings them to a leaderboard screen. The user enters their name and adds their score to the leaderboard. Then the user can start over and sing another time.

How we built it

Our code is structured by splitting features into different files for organization. We have GUI, Spleeter, Lyrics, Sound, Record, and Comparer. The GUI is the user interface created with Tkinter and CustomTkinter and calls methods from each of the other files when buttons are pressed and inputs are uploaded/typed. Spleeter splits the audios into the instrumental part and vocals. Crepe detects and analyzes the frequencies of vocals. Pygame allows the audio to be played. Pyaudio/scipy reads the audio. Numpy/statistics processes data.

Challenges we ran into

Because we used multiple tools that rely on Tensorflow, we ran into problems with interference. To solve this, we ran some tools in different processes using multiprocess, which also helped with performance issues. Additionally, we had problems with background noise so we implemented multiple filters on the data extracted from the audio. For example, after taking the difference between the two lists of frequencies, we removed any outliers outside of 2 standard deviations from the mean from the list of differences. This would lessen the impact of any extreme noises made during recording while still preserving the most singing.

Accomplishments that we're proud of

We were successfully able to implement multiple python libraries despite relative inexperience with the language. We are also very proud of the graphical user interface (built with custom tkinter and tkinter), which is very welcoming to all users and very easy to use with little bugs. Furthermore, the audio analysis using multiple AI tools is scaled well, and we are proud of the fact that it can differentiate good singing from bad singing.

What we learned

We learned how to use many new packages and libraries to integrate everything in a graphical user interface. We learned how to use Tkinter as well as CustomTkinter to create a user interface that is easy to use and aesthetic.

What's next for Karacroakie

We could upgrade the vocal analysis capabilities to include advanced metrics, such as vibrato measurement to provide users with more detailed feedback on their singing performance. Additionally, implementing user profiles would allow users to track individual progress, scores, and practice history. Introducing customizable challenges would engage users by offering them the opportunity to create or join competitions with specific objectives, such as achieving the highest accuracy or mastering a particular song. Lastly, developing a comprehensive song library with lyrics and predefined practice sessions would make the app more efficient to use.

Built With

  • crepe
  • lyrics.ovh
  • pyaudio
  • python
  • spleeter
  • tensorflow
  • whisper
Share this project:

Updates