Aliens in the Sky

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Inspiration

"Aliens in the Sky" is rock song made to drive to. Created for my video game, Jack's 510 Racing.

1. Core Concept and Origin

The song was originally composed for a driving game, specifically to accompany a mode where the player must escape a UFO while driving a car.

2. Musical Evolution

The song's final identity evolved to fit with the other music I have used in my game:

• Original Element: The track initially featured a creepy piano sound.
• Final Change: This was changed to a rock guitar sound to maintain a consistent high-energy, rock-oriented feel across the entire soundtrack.

What it does

The "Aliens in the Sky" music video functions as a short film that visually tells a rags-to-riches story. It depicts the experience of a band that was failing on Earth and then achieves massive success by transforming into the most popular band in the universe with an extraterrestrial audience.

How I built it

Phase 1: Music Composition and Style

The process began with the creation of the musical track:

• Initial Draft: I first generated a base electronic track using Suno.
• Style Refinement: I then remixed this track, influencing its style using other musical compositions I had created for my game, Jack's 510 Racing, to achieve the desired sound.

Phase 2: Image and Video Asset Generation

To create the visual components, I utilized a combination of leading AI models and significant local computing power.

1. Image Generation

• Tool Used: Gemini
• Method: I used Gemini to generate the base images for the video clips. This was typically done by providing a reference image and then instructing the model on specific aesthetic or compositional changes I required. Then I used Magic Eraser or Qwen Image Editor to remove the watermark.

2. Video Clip Generation

Most of the video clips were rendered locally. 2 clips with 8 seconds of time used Flow from Google.

Component	Tool/Hardware	Model Used	Parameters	Est. Time for 5 Seconds
Non Human Clips	NVIDIA GeForce 3080 (10 GB VRAM)	Wan2.2	5 Billion	approx 5 minutes
High-Fidelity Clips	NVIDIA GeForce 3080 (10 GB VRAM)	Wan2.2	14 Billion	approx 30 minutes
Other Clips	Google flow	N/A	N/A	N/A
Lip-Sync Video	NVIDIA GeForce 3080 (10 GB VRAM)	Hunyuan Video Avatar	14 Billion	Over 40 minutes

Note: The computational cost for high-resolution, complex video generation was substantial, with the most detailed clips, such as the lip-sync video, requiring over 40 minutes of processing time for every 5 seconds of output.

Phase 3: Editing

• Tool Used: CapCut
• Process: I used CapCut to edit all the rendered clips together, stitching the various segments (including the Wan2.2 clips, Google Flow clips, and Hunyuan Video Avatar lip-sync sections) into the final cohesive music video.

Challenges I ran into

My initial plan was disrupted when I decided to test Google's AI capabilities for video generation.

1. Credit and Resource Constraints

• Rapid Credit Depletion: I ran through my monthly free test credits in under an hour.
• The Pivot: This necessitated researching AI video models that could be run locally.

2. Technical Limitations and Time Cost

I ultimately decided to use the Wan2.2 and Hunyuan Video Avatar models, which introduced significant time-related challenges:

• Excessive Rendering Times: The primary problem was the long duration required to render even a small video clip using these models.
• Consistency Issues: A major challenge was maintaining visual similarity between clips for characters and objects.
• Stylistic Control: Achieving a realistic look for the aliens proved difficult, as the AI models often defaulted to a hand-drawn or stylized aesthetic instead of the desired realistic appearance.
• Mitigation Strategy: To prevent wasted time and computational effort, I had to ensure I developed an exceptionally good starting image and prompt before initiating any lengthy rendering job.

Accomplishments that I'm proud of

• Completion of a Complex Project: Successfully completing the entire music video from initial concept and music generation to final video rendering. Given the significant technical challenges, long rendering times (up to 40 minutes for 5 seconds of footage), and the quick pivot required after exhausting cloud resources, the successful delivery of the final product is a major accomplishment.

What I learned

A major learning outcome of this project was gaining practical knowledge about the available open-source AI models and understanding the real-world limits of my local hardware.

1. Model Availability and Compatibility

I learned that several cutting-edge AI models are accessible for local deployment, including those I used:

• Wan2.2 (Alibaba)
• Hunyuan Video Avatar (Tencent)

2. GPU Limitations and VRAM Management

With my NVIDIA GeForce 3080 (10 GB VRAM), I learned what my PC could realistically run:

• 10 GB VRAM: This is sufficient for most Stable Diffusion XL (SDXL) image models and smaller video models (like the 5 Billion parameter version of Wan2.2).
• The VRAM Bottleneck: I experienced the limits of my 10 GB VRAM when running the large 14 Billion parameter video models. While they could run, the VRAM constraint resulted in the long, multi-hour rendering times for short clips. This taught me that VRAM capacity directly correlates with rendering speed for large models.

What's next for Aliens in the Sky

The immediate future for "Aliens in the Sky" is finishing the game it was originally conceived for: Jack's 510 Racing.

Built With

• capcut
• flow
• gemni
• hunyuan
• suno
• veo
• wan2.2