BlackBox | Arty 2.0

A voice-directed, visually aware digital performer inside real captured & generated 3D worlds — with a live AI-VFX broadcast lane.

Comment

Inspiration

I have been building BlackBox, an AI-native filmmaking platform for directing virtual performers and creating cinematic scenes inside real-time 3D environments.

When ARDY was released, I immediately wanted to experiment with it. BlackBox already includes work involving Komodo, MotionBricks, and ProtoMotions, so ARDY fit naturally into the character and animation systems I was already developing.

Rather than treating the hackathon as a disconnected weekend project, I built ARDY 2.0 as a standalone BlackBox module. This allowed me to explore a brand-new animation system while continuing to build technology that can become part of the larger BlackBox platform after the event.

What it does

ARDY 2.0 is a voice-directed, visually aware digital performer that operates inside captured and generated 3D worlds, with a live AI-VFX broadcast pipeline.

The character can:

  • Stream real-time motion from the ARDY animation system
  • See the virtual environment through a camera mounted from its own point of view
  • Communicate voice-to-voice through Gemini Live
  • Receive spoken performance direction
  • Move through the environment and hit directed marks
  • Perform and record multiple takes
  • Send its program feed through Decart Lucy 2.5 for live, in-camera AI-VFX

The goal is to make directing a virtual character feel closer to directing a performer on a physical set. Instead of manually animating every movement, the filmmaker can communicate with the character, give it direction, observe the result, and record a usable take.

The system also separates the workflow into multiple views: a clean real-time 3D feed, a transformed Lucy 2.5 VFX feed, and a control interface for directing and monitoring the performance.

How I built it

I developed ARDY 2.0 as an independent module that can later be integrated into BlackBox.

The project combines several real-time systems:

  • ARDY provides the character animation and motion foundation
  • Gemini Live provides real-time voice conversation and spoken direction
  • A character-mounted virtual camera gives the performer visual context
  • The 3D runtime handles the character, environment, cameras, movement, and performance view
  • Decart Lucy 2.5 transforms the outgoing program feed with live AI-generated visual effects
  • A recording layer captures performances so takes can eventually be transferred into the BlackBox sequencer

The module was structured so that the motion system, voice model, vision system, VFX provider, and character representation remain separate. This is important because BlackBox is designed to remain model-agnostic as these technologies continue to improve.

The prototype was built using a combination of TypeScript, Python, PowerShell, WebGPU, and web-based real-time 3D technology.

Challenges I ran into

The largest challenge was coordinating several real-time systems within the limited timeframe of a hackathon.

Animation, voice, vision, rendering, recording, and AI-VFX all operate at different speeds and through different interfaces. Connecting them is not simply a matter of calling several APIs. They need to share a consistent understanding of what the character is doing, what the user has requested, and what should appear in the final program feed.

Latency was especially important. A virtual performer does not feel alive if it takes too long to respond to direction, and the Lucy 2.5 output needed to remain close enough to real time to function as a live VFX monitor.

ARDY was also extremely new, so there were fewer established workflows and integration examples to build from. I had to approach it as both a creative experiment and an engineering prototype.

Another constraint was limited API runtime. I only had a small amount of Lucy 2.5 generation time available, so the system had to be tested carefully without consuming the credits reserved for the final demonstration.

Finally, scope was a constant challenge. The long-term BlackBox vision includes character memory, spatial reasoning, physical interaction, motion editing, sequencer integration, and production-quality recording. For the hackathon, I had to identify the smallest version that could still communicate the larger idea.

Accomplishments that I'm proud of

I am proud that the project turns a newly released animation system into something designed specifically for filmmakers.

ARDY 2.0 is not only a character animation test. It connects motion, vision, voice direction, real-time rendering, performance capture, and live generative VFX into one coherent production concept.

I am also proud that the work is reusable. The hackathon module was intentionally designed as part of the larger BlackBox architecture rather than as a disposable demo. The research, integrations, and interface decisions can continue to support the main platform after the event.

Most importantly, the project demonstrates a different relationship between filmmakers and digital characters: the character can be directed as a performer instead of being treated only as an object that must be manually animated.

What I learned

The main lesson was less about discovering a new algorithm and more about validating ARDY as a practical foundation for real-time digital performance.

ARDY is a compelling system, and working with it reinforced how quickly character animation is moving toward interactive, directable performance. The animation model itself is only one part of the solution, however. The real value comes from orchestrating animation with voice, vision, spatial awareness, recording, and filmmaking tools.

I also learned that building ARDY as an isolated module was the right decision. It allowed me to experiment quickly without destabilizing the larger BlackBox application, while still producing work that can later be integrated into the platform.

The project further confirmed that live generative VFX can function as a separate broadcast layer. The clean 3D scene remains available, while the director can simultaneously view an AI-transformed version of the performance.

What's next for BlackBox | ARDY 2.0

After the hackathon, ARDY 2.0 will be integrated into the main BlackBox platform.

The temporary demonstration character will be replaced with the parametric character system I developed for BlackBox, allowing users to create and direct their own customizable performers rather than being restricted to a single character.

The next development stages include:

  • Recording performances into the BlackBox cinematic sequencer
  • Converting live performances into editable animation and keyframes
  • Adding persistent memory for direction, notes, and performance adjustments
  • Improving spatial awareness and object interaction
  • Supporting multiple characters and coordinated performances
  • Connecting ARDY with BlackBox's Komodo, MotionBricks, and ProtoMotions systems
  • Expanding support for captured environments, generated worlds, and Gaussian splat scenes
  • Making the live AI-VFX layer provider-agnostic
  • Developing a production-ready virtual filmmaking workflow

The long-term goal is a complete virtual production system where filmmakers can create a world, place a character inside it, speak to that character, direct a performance, record multiple takes, and immediately transform the result with cinematic AI-VFX.

Built With

Share this project:

Updates