Our combined passion for computer science and music drove us to the automation of an entirely AI-based DJ. Synthesizing its own music, PhantomDJ has trained itself to rap using a large dataset of lyrics from a variety of different rap artists. We included some of our favourite childhood artists such as Kanye West, Jayz, and Nas while also providing inspiration to those passions is music creation.

What it does

PhantomDJ is an AI-powered tool that can make your favourite rappers sing whatever you want. You select a famous rapper then you can provide the lyrics or the lyrics can even be autogenerated by the AI. After which the platform provides you with audio of your favourite rapper singing your lyrics in his voice. You could potentially drop Kanye West's new album that too on time XD.

Summarised User Workflow - user chooses artist > user can choose to input some lyrics > ai generates a verse and pulls a random beat for background music from youtube > the files are merged together to make one file > audio file is played on the frontend

How we built it

Building the app was a whole complex process. Let's divide it into 3 parts and understand it step-wise.

Our Stack stack

  1. Frontend - We coded the frontend in HTML/CSS/JS and we used Angular js as our framework which helped a lot in integrating the backend and the AI Model. The project was initially designed the project in Figma.

  2. Backend - Our main language in the backend was python. We had a flask server and hosted the backend on GCP. We used firebase for file storages (lyrics/audio files).

  3. Artificial Intelligence - We used Tensorflow to code our AI model which generates lyrics and the rapper's vocals. The model is based and built upon a Recurrent neural network (RNN). We trained the RNN model to come up with the lyrics given a large dataset of lyrics based on the songs the artist has made in the past Click to view dataset. As of now, there are 8 different rappers whose vocals can be generated with the given lyrics. And they are:

1.Kanye (0.8-0.9)

2.Nas (0.85-0.9)

3.Biggie (0.9-1.0)

4.Jayz (0.8)

5.Ross (1.0)

6.Kendrick (1.0) (No affected)

7.50 Cent (Not affected)


More About Our Recurrent Neural Network - We created the proprietary script that runs and trains the AI which uses recurrent neural networking. We chose to use a recurrent neural network over other neural networks because of its dynamic capabilities which we utilized in our voice synthesis. Additionally, the continuous change in vocal tonality while rapping could more easily be mimicked using this network.

This is how our RNN/AI Model works This is how our RNN/AI Model works

Challenges we ran into

ShellHacks 2021 gave us an amazing experience to work with friends from different countries. A path without problems is never interesting. This hackathon journey was very interesting and memorable to us with many challenges.

  1. We brainstormed and came up with an awesome idea - PhantomDJ. We had to work on various tech stacks required to successfully build this project. Our team included members with expertise in most of those skills. But, we lacked some mutual tech stacks like Angular for one of us and flask for another one. So it was hard for individuals of us to learn new technologies in a short time to work with the team on the team's tech stack but we eventually coped with it

  2. We used multiple technologies for developing this project. We had to individually build modules, integrate them with each other, and test them. These three stages were done multiple times and we came across many bugs and issues in our code which we debugged together and to make our code efficient and effective.

  3. Completing this project in a given duration with our teammates from different time zones worried us initially. But, the efforts took by the entire team to work restlessly even at night to make our team's idea into this successful project helped to overcome this challenge so easily.

Apart from learning and building a project, the challenges and experience that this hack gave us is the most valuable thing. Thanks to ShellHacks Team for organizing this amazing event.

Accomplishments that we're proud of

  1. We are proud of completing such a complex app in a limited time
  2. Building a Functional Artificial Intelligence-based voice synthesizer and getting so accurate results like near-real vocals
  3. Having such a great user interface
  4. Combining vocals with an appropriate background music
  5. Using AI to generate Lyrics

What we learned

We learned a ton during the event. Let's list some of them

  1. Integrating flask backend with angular.js
  2. Making AI models with tensorflow.js
  3. Using neural networks to generate lyrics
  4. How to synthesize text to voice
  5. Team coordination skills

What's next for PhantomDJ

Due to limited time, we weren't able to implement some of our planned features

  1. Having a deep fake/animated video of a rapper singing your lyrics using deep learning
  2. Adding more rappers
  3. Adding more music controls and features like autotune

We had a great time building this project and learning from shell hacks. For PhantomDJ, the whole codebase was entirely written by us in the given time duration


Built With

+ 25 more
Share this project: