Inspiration
I listen to audiobooks to be able to read while still getting work done, but found that many apps that can make custom audiobooks often feel robotic.
What it does
The app can take e-books and convert them into dynamic audiobooks, with unique voices for each character, background music, sound effects, and images. Users can upload an e-book and the app will analyse the book and create a Character and Chapter Bible. The Character Bible keeps a consistent description of the character's personality and design, which are then used to generate a consistent voice for the character and to ensure that images keep their design consistent. The Chapter Bible is used so that users can ask questions about the story, and the model will respond using only information that the user has already heard to avoid spoiling the story.
How we built it
Each aspect of the process (segmenting the story into sections like author notes, chapters, scenes, etc ; assigning dialogue to specific characters ; analysing characters to determine a voice and physical appearance ; generating and animating images of characters with consistent designs ; adding music and sfx) is attributed to a different model. Each is done separately, then are all added together.
Challenges we ran into
One issue was in regard to hardware limits, as I was running the models locally. This frequently led to crashes due to over usage of vram, but I was able to fix this by implementing safeguards at the cost of slower run time.
Accomplishments that we're proud of
The Chapter Bible was difficult, as it would frequently either not properly retrieve the information or would get information far in advance.
What we learned
I have never worked on a project that required multiple models, as I rarely needed them for specific purposes. This project helped me appreciate how specialised these models are.
What's next for Dramatize
Currently, the biggest slow down is that the app runs locally, which means that for lower tier hardware it is almost impossible to run. I would like to use a server with higher quality hardware to be able to run this, as it would also make the app smaller since users would no longer need to download the models themselves.
