GMM: Visual Scene Pipeline & OpenUSD Studio

Artist-led visual scene pipeline bridging physical collage, OpenUSD 3D spatial dioramas, and local-first AI agents via Model Context Protocol.

Comment

Inspiration

Gondurastration is my 20-year analog collage art practice. In physical collage, every scrap of paper, aged print, cut edge, and ink bleed carries tangible history. A single torn contour or placed silhouette suggests a human face or an emotional narrative without spoon-feeding it to the viewer.

As digital tools and generative models exploded, I observed a fundamental creative crisis: modern diffusion models produce flat, un-inspectable bitmaps with zero memory of their physical origins, zero editable spatial depth, and zero respect for provenance.

We created Gonduras Meat Market (GMM) to ask a transformative question: How can a physical collage evolve into an interactive, 3D inhabitable scene while preserving its material truth, spatial layering, and the artist’s explicit intent?

What it does

GMM: Visual Scene Pipeline is an end-to-end artist-led studio workflow that translates 2D collage art into layered 3D spatial environments:

  1. Physical DNA & Layer Decomposition: High-resolution collage scans (31 master collages in gondurastration_high/BEST and 15 narrative frames in HE HER) are segmented into individual physical layers (12 layered cutout elements in 3_TR_3—crown, monument, arrow, head, textures) mapped to true millimeter coordinates (1 px = 1 mm).
  2. OpenUSD Scene Construction: Elements are compiled into non-destructive Pixar OpenUSD (.usdc) stages across 10 canonical stages where physical cutouts exist as layered 3D planes with realistic depth, lighting, and camera paths.
  3. Agentic Orchestration via Model Context Protocol (MCP): Rather than relying on rigid, monolithic software, the studio is driven by AI agents communicating over 4 self-hosted Model Context Protocol (MCP) servers (gmm-core, gmm-board, gmm-project, gmm-gpu), allowing natural language queries to inspect scene assets, monitor GPU VRAM, and verify provenance.
  4. Workshop Window UI: A responsive, browser-based visual cockpit (http://localhost:7860) that renders real-time collage contact sheets, spatial bounding boxes, and GNV (Generative Neural Vault) sidecars.

How we built it

  • Amazon Bedrock (Multimodal Cognitive Council): Serves as our upstream director. We route vision-language queries and tool calling through Bedrock (Claude 3.5 Sonnet and Amazon Nova) via boto3 and LiteLLM to analyze reference artwork, evaluate compositional vectors, and generate structured parameter sidecars.
  • Local Agent Execution (GONVEX / Hermes): A bare-metal agent runtime operating within a strict 12GB VRAM envelope on an NVIDIA RTX 3060 workstation, executing local PyTorch pipelines, C++/Slang shader passes, and Docker containers without memory overlap.
  • Storage & Provenance (AWS S3 & GNV): Scene manifests are sealed into .gnv.json sidecars containing model checkpoints, seeds, control adapter parameters, and SHA-256 asset hashes. Amazon S3 serves as our persistent remote vault for scene manifests and multi-pass latents.
  • Spatial 3D Bridge: Python and OpenUSD tools assemble extracted silhouettes and depth slices into layered 3D dioramas with decoupled UV material bindings.

Challenges we ran into

  1. From Flat Scans to Spatial Depth: Extracting spatial relationships from flat 2D collage scans is non-trivial. Inferred depth must remain an editable hypothesis until confirmed by the artist. OpenUSD proved essential because it lets us adjust layer offsets in millimeters without baking them into pixels.
  2. Autonomous Tooling & Cloud Guardrails: During experimental testing with cloud agent toolkits, we experienced firsthand how rapid automated inference can exhaust promotional credits and trigger unanticipated charges. We had to design strict local VRAM and API guardrails, decoupling the core creative pipeline to run 100% local-first.
  3. Preserving Ambiguity in Semiotics: Standard AI vision taggers tend to oversimplify complex visual metaphors. In Frame 03 of our HE HER series, an upward yellow collage arrow represents ego ambition, while surrounding blue triangles suggest off-frame subjectivities. Training our pipeline to honor this ambiguity rather than flattening it was our greatest conceptual hurdle.

Accomplishments that we're proud of

  • Compiling fully inspectable, 10-stage OpenUSD scene hierarchies from real analog collages with 100% resolved asset dependencies.
  • Deploying a live, working hybrid architecture: Bedrock handles high-dimensional reasoning and parameter extraction in the cloud, while our local workstation executes sub-gigabyte spatial passes and 3D diorama staging.
  • Maintaining complete material truth: every digital asset links back to its verified analog scan hash in the GNV sidecar.

What we learned

Offloading multimodal reasoning to Amazon Bedrock transformed our pipeline architecture. Instead of burdening local workstation memory with large vision-language models, Bedrock acts as an on-demand technical director that returns clean, structured scene parameters in seconds, preserving our local GPU capacity exclusively for real-time spatial compilation.

What's next for GMM: Visual Scene Pipeline

  • Releasing the open GNV specification v1.0 and native OpenUSD export plugins for Blender 5 and Unreal Engine 5.
  • Expanding our OpenUSD stages into interactive VR/XR environments for living room exploration via Fire TV and WebXR.
  • Connecting our spatial scene representations to Isaac Lab / Isaac Sim robotics pipelines to test physical collage characters in dynamic physics environments.

Built With

Share this project:

Updates

Submission history