RoBlocks

Inspiration

Traditional filmmaking is plagued by the "Fix it in Post" fallacy, where indecision on set leads to massive production burns averaging $250k daily. Traditional blocking consumes up to 40% of a production day because adjusting physical actors and camera rigs is slow and prone to human error. We realized that while pre-visualization (previs) has gone digital, there is still a massive gap between a 3D render and the physical reality of a set. Our inspiration was to "Fix it in Pre" by moving the heavy labor of blocking into an immersive robotic rehearsal phase. We wanted to give directors a way to "dream" a scene into motion using robots as high-fidelity physical avatars. As part of our exploration of machines dreaming, we created a short immersive film in DeoVR using our blocking application. The film follows a robot living a monotonous routine, identifying objects inside a small enclosed room each day as part of its training loop. When a knock draws it outside for the first time, the robot encounters a world it was never designed to understand. Unable to classify what it sees, it begins to imagine rather than label, slowly finding beauty through misinterpretation and curiosity. This film became both our narrative inspiration and a technical proof of concept for how our system can block, rehearse, and visualize scenes where machines learn to dream beyond their programmed boundaries.

What it does

RoBlocks is a robotics-powered previs system that allows directors to cast robots, rehearse in VR, and shoot with physical certainty. The system acts as a "Robotics as a Service" (RaaS) platform where directors can:

Direct by Interaction: Use VR hand-tracking and voice commands like "Come closer" or "Take two steps" to sculpt a robot's performance in real-time.
AI Scenario Generation: Feed scripts and emotional beats into an AI-agent interface that envisions movement and timing.
Unity Digital Twin: Access a 1:1 digital twin that mirrors every physical movement, allowing for frame-by-frame articulation adjustments of the robot's limbs.
Humanify Motion: Achieve organic sequences that can be repeated 1,000 times with zero deviation, reducing on-set production time by an average of 30%.

How we built it

The technical backbone of RoBlocks bridges high-level AI dreaming with low-level robotic precision. We spent 48 hours architecting a complex stack that flows from ROS to C++, then through low-level and high-level Python controls to manage the Booster robot.

The Brain: We integrated an onboard Ollama instance to process scripts, allowing the robot to "imagine" actions based on textual emotional beats.
The Twin: Joint positions and velocities are streamed over TCP to a Unity environment. This digital twin allows for fine-tuning poses which are then replayed back to the physical hardware.
The Interface: We utilized the Meta SDK and OpenXR to implement hand tracking. We one-hot encoded specific hand gestures to translate spatial virtual anchors into direct robotic motion, giving the director an immersive "flick of the wrist" control over the scene.

Challenges we ran into

The primary hurdle was the communication latency and stability of the low-level controls for the Booster robot. Debugging the multi-language pipeline (ROS > C++ > Python) required significant effort to ensure real-time responsiveness. Additionally, refining Meta Hand Tracking to be precise enough for "sculpting" a performance avoiding jitter while maintaining intuitive gesture recognition proved to be a major hurdle during the 2-day build.

Accomplishments that we're proud of

We are incredibly proud of achieving end-to-end real-time control over TCP, creating a seamless link between the director's physical gestures and the robot's response. Successfully implementing the Ollama multi-modal voice/text control allows the robot to act as a true creative partner rather than just a machine. Finally, our gesture encoding to motion execution pipeline demonstrates a future where human creativity and robotic precision are perfectly synchronized.

Proof of Concept — Short Film Production

As a proof of concept, we produced a VR180° 3D short film using our robotics-powered VR system developed during the hackathon. The modular robot-block architecture enabled rapid camera and actor blocking, real-time iteration, and seamless integration with the DeoVR streaming platform. Leveraging this workflow, we gathered footage and delivered a finished short film in under 48 hours. A teaser is linked below and can be viewed on the DeoVR platform or natively on Meta Quest using VR180° 3D settings.