A video model does more than recognize what is in a frame. To predict what comes next, it must build internal representations of how a scene changes over time. Those states can encode motion, viewpoint changes, object dynamics, and the effects of actions. A camera moving forward, rotating, or closing in on an object is not far from a robot doing the same thing with cameras mounted on its body.
This project tests whether those internal dynamics can be reused for control.
We freeze the MiniMax H3 video model and use it as a temporal visual feature extractor. The inputs are a task instruction plus synchronized external and wrist-camera observations from LIBERO. A separate roughly 255M parameter Transformer reads six selected H3 layers and predicts 34 consecutive 7D robot actions.
The policy alternates cross-attention and self-attention blocks. Causal video attention, historical cross-attention cutoffs, and isolated action-query groups prevent future observations from leaking into predictions.
Current training uses LIBERO-Spatial with 450 demonstrations for training and 50 for validation.
Built With
- torch
Log in or sign up for Devpost to join the conversation.