Motion Cast

You video AI generation will be easier with Motion Cast.

Comment

Motion Cast: Directing Motion Before Generating Video

About the Project

Motion Cast is a tool for AI video generation. It helps a creator direct a short performance before asking a video model to render the final character, style, and world.

The workflow starts with a timed action plan such as: jump, walk forward, turn, backflip. ARDY generates one continuous skeletal performance from that plan. Motion Cast then uses Blender on the creator's own machine to apply camera direction and render a neutral MP4 motion reference. That MP4 can be combined with a character sheet and an art-direction prompt in Seedance 2.0 to create the final video.

The point is not to replace AI video generation. It is to give it a better brief. Think of it like a motion template.

Inspiration

Current AI video tools can produce striking single shots, but keeping motion, framing, and a character's identity coherent across several actions is still difficult. A prompt like “a person jumps, walks, then backflips” describes what happens, but not the timing, the continuity, or the camera language.

We were inspired by the production workflow of previsualization: block the performance first, direct the camera second, and add the finished character and style last. Motion Cast adapts that idea for a creator who does not need to animate a rig manually or learn a full 3D package before they can make a useful reference.

How We Built It

Motion Cast is a local React control room backed by FastAPI.

  1. The creator edits a 15-second timeline with separate Action and Camera tracks.
  2. The app sends the contiguous action timeline to the ARDY Hugging Face Space through /generate_timeline_motion_package.
  3. ARDY generates each action autoregressively from the prior action's state, returning one continuous .npz motion package instead of disconnected clips.
  4. Motion Cast invokes its bundled Blender renderer locally. It applies a camera plan per scene, renders a blue or green proxy performer, and saves both the Blender scene and PNG frames.
  5. FFmpeg encodes the frames into a broadly playable H.264 MP4.
  6. The creator previews and downloads the MP4 as a Seedance motion reference.

The editor is built with React, Vite, and DesignCombo's timeline canvas. FastAPI manages the ARDY request, Blender process, and job polling. The Blender renderer is included in the repository, so a developer can reinstall Motion Cast without cloning the full ARDY source tree.

What We Learned

We learned that camera cuts cannot be treated as simple metadata. For a reliable motion reference, the camera must be animated frame by frame in Blender, with each scene's selected movement driving the same render camera. This makes a side track feel different from a follow shot even when the body motion is identical.

We also learned that an API-first motion model is more useful than a beautiful isolated demo. ARDY's continuous timeline endpoint makes it possible to hand a renderer structured motion data, while a local MP4 gives downstream video models a compact, familiar reference format.

Finally, the project reinforced a small but useful principle: AI video workflows become more approachable when the creator can work in familiar film language. “Dolly in for the reach” is easier to reason about than a long prompt trying to describe both a movement and a lens decision at once.

Challenges We Faced

Continuous motion. Generating every action separately created visible discontinuities. We solved this by using ARDY's timeline endpoint, which carries its internal motion state from one action to the next.

Per-scene cameras. Our first camera implementation applied only the last selected camera behavior. The fix was to calculate the active scene for every rendered frame and keyframe one camera through the full timeline.

Portable MP4 output. Blender's video support changes between versions. We render PNG frames first, then use FFmpeg to create an H.264, yuv420p MP4 that plays consistently in browsers and video-model upload flows.

Making 3D direction approachable. Blender is powerful but intimidating. The interface reduces it to an editable action sentence, start and end times, and a small set of cinematic camera moves: locked-off, follow, side tracking, dolly, orbit, crane, and handheld.

A deployable ARDY API. We converted the ARDY Space from a full interactive playground to an API-first Space with public-facing documentation.

What's Next

The next version of Motion Cast will add shot templates, timeline playback synchronized to rendered previews, character-sheet generation, and a final Seedance handoff that packages the character sheet, reference MP4, and scene prompt together.

Our larger ambition is simple: let more creators direct AI video with better workflows.

Built With

Share this project:

Updates