Sanctuary Super Heroes Trailer

Inspiration

Partially inspired by the Book series, Dungeon Crawler Carl

How we built it

Sanctuary is almost entirely AI-created. I largely wrote the script, but songs, music, sound effects, images, and video are generated with AI across a wide range of programs, the primary ones being Fal AI for almost all the art and animation, and 11 Labs, which was used to create a lot of the audio.

The basic workflow is as follows: first, the script and songs. After this, we get a reference picture for each character, as well as a file full of text descriptions for scenes and characters, and generate a still. This is then often edited before we convert it to a video. We'll iterate on prompts recursively until we arrive at outcomes we like. Often, video clips are cropped or adjusted, and sometimes we use more extreme editing techniques. Sometimes we also use start and end frames when generating.

Challenges we ran into

The main difficulties are maintaining character consistency and making fast-paced action scenes work with multiple characters. Especially, though, I find writing dialogue hard.

Accomplishments that we're proud of

The video we've produced is at least on par with the animation quality of more children's cartoons on TV and most anime. It would likely have cost 100,000+ dollars to create before AI. Now it can be made for less than $1000, likely resulting in more than a 100x cost saving.

Also, all the songs are original creations for a non-musician; this makes me very proud.

What we learned

I've been working on an AI video for about 8 months now. Almost all the previous projects were entirely musicals, though, so we learned a lot working with lip-syncing and adding sounds with 11 Labs.

What's next for Sanctuary: Super Heroes Trailer

We hope to continue the series and have already started work on episode 2.

Built With

• 11labs
• fal-ai