Inspiration
I wanted to tell a story about two strangers who connect through a song. My hope was to do this in an emotionally believable way using AI video.
What it does
The Album tells a simple story of one man who is haunted by a song of heartbreak. He plays the song from an old album then goes on a walk to cope with his emotions. In an empty train station, he hears a lonely street performer playing the same song. Their brief interaction reveals a surprise for each.
How I built it
- Veo 3 in Flowfor text-to-video, including experimenting with advanced prompt techniques for consistency and cinematographic control.
- Kling for lip-syncing... or really lip-singing.
- Suno for generating the original song, the subway busker’s cover version, and their duet.
- Nano Banana for image-to-video start frames and object removal.
Challenges I ran into
- Emotional storytelling can be difficult with AI. I focused on how their brief interaction might transform the emotional state of both characters.
- Long, single-take shots can be an effective way to convey emotional shifts but that requires a lot of creative problem-solving to maintain control over the course of a 20 second shot.
- Maintaining consistent characters and a consistent cinematic style can be very difficult in text-to-video so tried a highly structured prompt style.
Accomplishments that I'm proud of
- The dialogue section between the two is a "single-take" 20 second shot. This required AI subject removal with Nano Banana, two start/end frame animations in Kling, and a 10 second timed dialogue in Veo that turned my English into Japanese.
- We hear the nostalgic vinyl version first, followed by the subway busker's cover. After their brief interaction, we hear a duet which represents their connection for the viewer, even though the duet does not exist in the world of the film.
- The rack focus between the characters as they speak was done entirely through prompting.
- Translating the lyrics and dialogue into Japanese, plus using Gemini to verify that my translations were correct
What I learned
- Better ways to structure prompts for consistency
- An advanced workflow for having more control over lip-singing
- Prompting for characters to speak in other languages
What's next for The Album
The subway street performer has plans to release a music video
Built With
- kling
- suno
- veo