Inspiration
Eric's idea. Image generation linked to Spotify uses lyrics to generate music video and create transitions between images.
What it does
Creates tiktokable music videos from song lyrics.
How we built it
Flask Server responsible for orchestrating video creation process:
- Creates prompts for all videos using GPT + lyrics + music analysis information
- Stitches together videos and adds effects/transitions
- Overlaying soundtrack over newly created music video
Technlogies Used
- Huggingface text-2-video model
- Huggingface video-2-video model (for upscaling) (not in demo)
- Modal + FastAPI (for running GPU-heavy video generation models in the cloud)
- Flask for backend, communicating with Modal API + frontend
- GPT-3 for video generation prompt creation
- Cyanite.ai for music analysis (mood, genre, energy, etc.) (not in demo)
- Moviepy for editing all music video components together into single video
- HTML/CSS + JavaScript for UI
Challenges we ran into
- Video: computationally intensive.
- Switch from Google Collabs to learn / use Modal to access Nvidia processing.
- Video Libraries to work together: Open CV / Moviepy
Accomplishments that we're proud of
- Text to Video Generation
- Branding / Aesthetic
What we learned
- Teamwork + git
- Movie editing Python libraries
- Modal
- Using Hugging Face models
- Eleven Labs
What's next for Ballad Bazaar
- Mobile Optimized
- Audio analyzer to further modify text-2-video prompt and to apply a color filter based on mood / sound / lyrics
- Allow user to specify aspect ratio of outputted video
Built With
- css
- flask
- graphite
- html5
- huggingface
- javascript
- modal
- moviepy
- openai
- python
Log in or sign up for Devpost to join the conversation.