AniMangaMusic

A site that automatically adds music to your manga that aligns with the mood of the story using the gemini api!

Comment

AniMangaMusic - Bridging the gap from manga to anime!

Inspiration

Have you ever read a manga and thought: The manga has better pacing but I wish I could listen to the amazing anime OST! or finished an anime and thought: Can't wait for season 2! I would read the manga, but then I'd miss out on the music.

I know I have. I've even tried reading comics while listening to music, but the music never lines up with the comic. Even if I put on Batman music while reading Batman, the music is often intense when the characters are just chilling!

Wouldn't it be great if there was a site that detected what page you were on and selected music that fit with the specific mood?

Yes. It is great.

What it does

  • Upload your comics and/or manga and read
  • Music selected by the gemini api corresponding to the mood of the page you are reading: fights have battle music, dramatic moments have dramatic music, etc.
  • Sound effects selected by the gemini api correspond to what is happening on the page: lasers make laser sounds, punches make punch sounds, etc.
  • Immersive glow behind the comic corresponding to the color of the page

How I built it

  • Made with python (flask), js, html, css
  • uses the gemini api to detect the most appropriate music and sfx for each page

Challenges I ran into

  • figuring out how to post images to a REST API

Accomplishments that I'm proud of

  • Correctly detects things like laser beams and punches right when they happen and adds appropriate sound effects
  • Thanks to ai, it works with any comic, even ones that I did not try during production
  • Is cost efficient, after using it all throughout the hackathon, it only used about 0.50$ because of the gemini api.

Could be even more cost-optimized if:

  • only changes songs on pages with major tonal shifts
  • uses a less powerful gemini model
  • compresses the images before sending
  • maybe even uses cheaper text detection api's before sending text to gemini
  • use a built-in LLM instead of an api to use a bit of the user's gpu instead of an API, to bypass the need for a subscription service.
  • It's actually cost-efficient enough that I plan on making it a full, publicly accessible website after my exams are done.

What I learned

  • How to post an image to a REST API.
  • To only really start with a minimum viable product in a hackathon: I didn't have to make it a flask app since I didn't have time for the features that needed python. I could have just put my api calls in javascript and saved time.

Planned Features

  • Find a better name for the site
  • Cost-optimize even further
  • Make it a publicly accessible site with it's own domain
  • Upload your own songs
  • Read panel by panel
  • Conversion to webtoon format for phones
  • Database with greater song and sfx variety
  • Android app

Credits

Notes

  • This devpost description was updated after the end of the hackathon.
  • Code is basically 99% human written: ai_use.txt in the github repo contains every ai prompt that was used in the making of this project, mostly just general syntax that happened to be faster to find with an ai overview.

Built With

Share this project:

Updates