SoundSketch: Create Comic Books From Your Favourite Stories

Inspiration

We realized stories are kind of stuck. Some people love reading, some people love listening, and some people only read if there are pictures involved. But stories usually come in one format, and that’s it.

What it does

A way for stories to go from audio --> comic and comic --> audio, depending on how you want to experience them. You can upload a story or audio, and by clicking a button, SoundSketch rebuilds it in the opposite format, retaining the same plot and characters, but presenting it in a different way to read/listen to it.

How we built it

Going from audio --> comic, we transcribed the speaking so that the JSON file can be input into Gumloop where AI nodes helped generate the final comic. From comic --> audio, any picture is uploaded which Gumloop then outputs as a voice (with the help of ElevenLabs).

Challenges we ran into

Keeping characters looking the same (harder than it sounds) and drawing their gender based on their name (and neutral if undetermined). Also making sure the stories still flowed after being converted while not losing important details!

Accomplishments that we're proud of

We actually made it work end-to-end!! Stories get turned into comics. Comics get turned into audio. We built something that helps accessibility and creativity (barely) on time.

What we learned

By making our product more accessible, it makes it a better website for everyone. We learned how to use Gumloop and ElevenLabs and found ways to incorporate them into our solution. Inevitably, we also learned how to debug our work – especially in new software we haven't seen before.

What's next for SoundSketch: Create Comic Books From Your Favourite Stories

More styles of drawing, more methods of reading (i.e. vertical scrolling) and adding more languages that SoundSketch could transform from audio book to comic and vice versa. Also, more specialization with voices/tones and differentiating characters from one another.