DJ Background Generator

Automatically create time-synced DJ-style lyric visualizations using Blender and generative AI

Comment

Inspiration

DJs, raves, etc. We want to be DJs but creating our own backgrounds is tedious.

What it does

DJ Background Generator creates music videos with time-synced lyrics and AI-generated backgrounds. It takes a song file and word-timed lyrics in LRC format as input, then produces a complete video using Blender where each lyric appears with a unique AI-generated surreal background synchronized to the music.

How we built it

There's two components to this project: the Blender-based video generation and rendering and the custom ComfyUI/ControlNet based background image render.

The GitHub repo has a pipeline that

  1. Parses LRC subtitle files to extract time-synced lyrics
  2. Generates a Blender project with text objects for each lyric
  3. Renders individual frames for each lyric line
  4. Submits rendered frames to a custom pipeline in ComfyUI for AI background generation
  5. Returns the frames and renders them into a music-synced background video with lyrics

Challenges we ran into

  • Achieving precise time synchronization between lyrics, audio, and visual transitions
  • ComfyUI is confusing, slow, and pretty expensive
  • AI's don't do a good job interacting with Blender

Accomplishments that we're proud of

Lots of scripts doing almost fully automated interaction with Blender!

Custom ControlNet + image pipeline, as a service!

What we learned

We gained experience integrating Blender's Python API for programmatic 3D rendering, working with ControlNet and image models, and figuring out how to subtitle and time-sync lyrics really fast.

Built With

Share this project:

Updates