DropBeat

• 

  World Model music Video

• 

  Screen

Inspiration

What if you could direct a music video inside an evolving AI world?

DropBeat was inspired by the human mind and world models. AI systems that generate an unfolding visual environment you can influence. Instead of describing everything upfront and waiting, we wanted creators to react as ideas appearedm introduce a character, add a tiger, or move the scene to the ocean.

The iPhone Duo’s spacious, adaptive layout made this interaction feel natural: pictures and controls beside the playing video. Our goal was to make music-video creation approachable even for someone who has never opened a video editor.

What it does

DropBeat is a world-model-powered music video studio.

Start with an opening idea and preload your pictures. While the video plays, drag in an image or type a direction to steer the evolving scene. The instrumental responds, while original English spoken-rap phrases follow the current direction.

Clear feedback shows whether each drop was received or accepted. Loading a picture does not introduce it automatically you decide when it enters.

After the take, DropBeat generates a polished vocal track and applies AI-directed Blender edits, including colour, motion, and English captions. You can preview the result, export it, or continue editing in Blender.

How we built it

We built a native SwiftUI app with an adaptive iPhone Duo layout, informed by Bitrig’s development guide. Native controls handle picture imports, drag-and-drop, prompt copy/paste, playback, and sharing.

Reactor/Orbis provides the live video world model. A WKWebView media engine connects through its JavaScript SDK and sends new directions to an ongoing generation session. Gemini and Lyria handle image interpretation, lyrics, spoken vocals, and music.

A local Next.js and TypeScript backend manages provider connections and exports. For finishing, AI proposes a structured edit plan that Blender executes through a trusted Python renderer. FFmpeg prepares the media, while RevenueCat provides optional checkout and entitlement checks.

Challenges we ran into

Steering a world model is probabilistic. An accepted direction does not guarantee an immediate visual change or an exact likeness. Maintaining character identity, scene continuity, and consistent visual quality was one of our biggest challenges.

We also had to coordinate video, instrumental music, and spoken vocals with different generation delays. Live vocals currently use reactive spoken rap; the polished musical performance is generated afterward.

Connecting native drag gestures to the web media engine required careful handling and clear feedback. Our finished showcase also includes additional AI-generated inserts where the live output needed refinement, distinguished from the actual simulator capture.

Accomplishments that we're proud of

Turned world-model interaction into a simple, picture-driven creative experience. Connected live scene steering, instrumental music, and reactive spoken rap. Built native picture imports, visible drop feedback, and prompt copy/paste. Recorded real simulator interactions and verified finished-video playback. Created an original one-minute rap video with English captions and an AI/Blender finish. Published a standalone repository with visual documentation, setup instructions, and an understandable architecture.

What we learned

World models introduce a different creative workflow: you can respond to an evolving scene instead of making every decision before generation begins.

That freedom needs clear feedback. Users should understand whether a direction was received, accepted, or still waiting to appear. A simple interface depends on careful coordination underneath.

We also learned that live generation and post-processing serve different purposes. Live generation encourages experimentation, while editing improves consistency, timing, and presentation. A 4K export alone does not guarantee high-quality footage—the underlying visuals and finishing process matter.

What's next for DropBeat

Our next priority is improving video quality and post-processing. We want sharper visuals, more consistent characters, smoother motion, and cleaner transitions as users steer the world model.

We plan to expand the AI-powered Blender workflow with better colour grading, beat-synced cuts, tighter vocal and caption timing, and higher-quality upscaling. The aim is to turn an experimental live take into a cohesive, shareable music video with minimal manual editing.

We also want to reduce response delays, add cue history and undo, improve recovery when providers disconnect, and validate the experience on physical devices.

The goal remains simple: direct the world live, then let AI help polish the result.

Start with an idea. Drop a picture. Direct the beat.

Built With

• bitrig
• blender
• gemini
• lyria
• openai
• orbis
• revenuecat
• swift
• uikit
• webkit
• xcode