Manga are Japanese comics, considered to form a genre unique from other graphic novels. Similar to other comics, it lacks a musical component. However, their digital counterparts (such as sites like Webtoons) have innovated on their take on the traditional format with the addition of soundtracks, playing concurrently with the reader's progression through the comic. It can create an immersive experience for the reader building the emotion on screen. While Webtoon’s take on incorporating music is not mainstream, we believe there is potential in building on the concept and making it mainstream in online manga. Imagine how cool it would be to generate a soundtrack to the story unfolding. Who doesn't enjoy personalized music while reading?
💻 What it does
- Users choose a manga chapter to read (in our prototype, we're using just one page).
- Sentiment analysis is performed on the dialogue of the manga.
- The resulting sentiment is used to determine what kind of music is fed into the song-generating model.
- A new song will be created and played while the user reads the manga.
🔨 How we built it
- Started with brainstorming
- Planned and devised a plan for implementation
- Divided tasks
- Implemented the development of the project using the following tools
Tech Stack : Tensorflow, Google Cloud (Cloud Storage, Vertex AI), Node.js
Registered Domain name : mangajam.tech
❓Challenges we ran into
- None of us knew machine learning at the level that this project demanded of us.
- Timezone differences and the complexity of the project
🥇 Accomplishments that we're proud of
The teamwork of course!! We are a team of four coming from three different timezones, this was the first hackathon for one of us and the enthusiasm and coordination and support were definitely unique and spirited. This was a very ambitious project but we did our best to create a prototype proof of concept. We really enjoyed learning new technologies.
📖 What we learned
- Using TensorFlow for sound generation
- Planning and organization
- Time management
- Performing Sentiment analysis using Node.js
🚀 What's next for Magenta
Oh tons!! We have many things planned for Magenta in the future.
- Ideally, we would also do image recognition on the manga scenes to help determine sentiment, but it's hard to actualize because of varying art styles and genres.
- To add more sentiments
- To deploy the website so everyone can try it out
- To develop a collection of Manga along with the generated soundtrack