Director3D

3D Layout Control in Text-to-Image Generation

Comment

Director3D: 3D Layout Control in Text-to-Image Generation

Inspiration

Current text-to-image (T2I) diffusion models (such as Stable Diffusion and FLUX.1) can produce breathtaking visuals, but fine-grained control over spatial layout remains a persistent bottleneck. While 2D bounding boxes or edge detectors (like ControlNet) allow rough placement, they completely disregard true 3D spatial geometry:

  1. Occlusion & Depth Ambiguity: Standard models struggle to understand which object is in front or behind when bounding boxes overlap.
  2. Perspective & Orientation Discrepancies: Controlling physical orientation (e.g., yaw/azimuth angles \(\theta\)) or metric scale (\(w \times d \times h\)) is nearly impossible with pure text prompts.
  3. Camera Dynamics: Changing camera elevation or lens focal length without distorting the scene requires 3D awareness.

We were inspired by recent breakthroughs in spatial conditioning—specifically bridging computer graphics and generative AI—to build an interactive studio where creators can direct a 3D scene with the precision of a film director or CGI artist, and have modern diffusion models faithfully synthesize the result.

What It Does

Director3D is an end-to-end interactive system that enables metric 3D bounding box conditioning for text-to-image synthesis using FLUX.1-dev:

  • Interactive 3D Frustum Layout: Direct placement of 3D oriented cuboids defined in metric camera space:

$$ \mathbf{B}_i = \left( x_i, y_i, z_i, w_i, d_i, h_i, \theta_i \right) $$

where \((x_i, y_i, z_i)\) represents the 3D centroid relative to the virtual camera, \((w_i, d_i, h_i)\) are metric dimensions, and \(\theta_i \in [-180^\circ, 180^\circ]\) is the azimuth rotation.

  • Occlusion-Aware Mask Decomposition: Instead of flattening bounding boxes into ambiguous 2D rectangles, our backend renders depth-ordered and exact projected segmentation masks \(\mathcal{M}_i \in {0,1}^{H \times W}\) along with perspective wireframes.
  • Subject-Specific Token Grounding: Each 3D box is bound to a distinct subject prompt description via placeholder token injection:

$$ \text{Prompt} = \text{"a photo of } \langle\text{placeholder}\rangle \text{ [subject}_1\text{] and } \langle\text{placeholder}\rangle \text{ [subject}_2\text{] in [surroundings]"} $$

  • Virtual Camera Control: Adjust camera elevation \(\phi\) and focal length \(f\) (in mm) in real time to simulate realistic wide-angle or telephoto perspective shifts.
  • Metric Proportions Harmonization: An automated scaling assistant that ensures realistic relative physical scale between real-world assets (e.g., dogs, armchairs, cars, tables).

How We Built It

1. Dataset Preparation & Synthetic 3D Data Engine

Training a diffusion model for metric 3D layout control requires massive, high-quality paired data with exact ground-truth 3D annotations, which does not exist at scale in natural image datasets. To solve this, we built a procedural 3D data synthesis engine:

  • 3D Asset Curation & Placement: We curated thousands of diverse, watertight 3D models spanning indoor furniture, animals, vehicles, and everyday objects with calibrated real-world metric scales (\(w \times d \times h\)).
  • Realistic Scene Generation: Objects were randomly composed in realistic spatial configurations, enforcing ground plane constraints (\(z \geq 0\)) and simulating diverse spatial relations (adjacent, stacking, depth staggering).
  • Camera Frustum Randomization: Virtual pinhole cameras \(\mathbf{K}\) sampled realistic viewpoints with random elevations \(\phi \in [0^\circ,60^\circ]\), azimuths \(\theta \in [-180^\circ,180^\circ]\), and focal lengths \(f \in [24\,\text{mm},85\,\text{mm}]\).
  • Multi-Modal Ground Truth Extraction: For every synthesized scene, our automated headless pipeline rendered:
    1. Photorealistic Target Image: \(\mathbf{I}_{\text{rgb}} \in \mathbb{R}^{H \times W \times 3}\).
    2. Perspective Layout Condition: A perspective wireframe render showing projected 3D cuboid edges and visible faces \(\mathbf{I}_{\text{layout}}\).
    3. Occlusion-Aware Visibility Masks: For each object \(i\), we rendered its exact unoccluded projection mask alongside its depth-tested, visible mask \(\mathcal{M}_i \in {0,1}^{H \times W}\), taking into account intervening objects.
    4. Structured Text Pairings: Dense scene captions combined with individual object prompt descriptions bound to specialized spatial tokens.

2. Model Architecture & Multi-Stream Attention Steering

Our generative backbone builds on FLUX.1-dev—a 12-billion-parameter rectified flow transformer comprising 19 Double-Stream Blocks (which model joint multimodal attention between image latents and text embeddings) and 38 Single-Stream Blocks (which concatenate image and text sequences into unified representations).

  • Mathematical Problem Formulation:

A scene with \(N\) objects is parameterized by oriented 3D bounding boxes in camera coordinates:

$$ \mathbf{B}_i = (x_i, y_i, z_i, w_i, d_i, h_i, \theta_i), \quad i \in {1,\ldots,N} $$

Projecting \(\mathbf{B}i\) through the pinhole camera matrix \(\mathbf{K}\) yields a perspective layout condition image \(\mathbf{I}{\text{layout}} \in \mathbb{R}^{H \times W \times 3}\) and binary per-object visibility masks \({\mathcal{M}i}{i=1}^{N} \in {0,1}^{H \times W}\).

  • Placeholder Token Grounding & Call IDs:

To guarantee that prompts never bleed across spatial bounds, we bind each individual object description to a dedicated \(\langle\text{placeholder}\rangle\) token in the sequence:

$$ \text{Prompt} = \text{"a photo of } \langle\text{placeholder}\rangle \text{ [subject}_1\text{] and } \langle\text{placeholder}\rangle \text{ [subject}_2\text{] in [surroundings]"} $$

During tokenization, each subject's text tokens are indexed by precise token ranges (\(\text{call_ids}_i\)).

  • Custom Multi-Stream LoRA Processors:

We implemented custom attention processors—MultiDoubleStreamBlockLoraProcessor across the 19 Double-Stream layers and MultiSingleStreamBlockLoraProcessor across the 38 Single-Stream layers (hidden dimension \(D = 3072\)). In every block:

  • Low-rank decomposition matrices (\(W = W_0 + \frac{\alpha}{r}BA\)) are injected directly into the projection heads (\(W_q, W_k, W_v, W_o\)).

  • For an input token sequence \(\mathbf{h}\), the update is computed as:

    $$ \Delta \mathbf{h} = \sum_{m=1}^{N_{\text{lora}}} w_m \cdot \left(\frac{\alpha_m}{r_m}\mathbf{B}m\mathbf{A}_m\left(\mathbf{M}{\text{cond}} \odot \mathbf{h}\right)\right) $$

    where \(\mathbf{A} \in \mathbb{R}^{r \times d_{\text{in}}}\) is initialized with Gaussian \(\mathcal{N}(0,1/r)\), \(\mathbf{B} \in \mathbb{R}^{d_{\text{out}} \times r}\) is initialized to zero, and \(\mathbf{M}_{\text{cond}}\) isolates condition channels.

    • Occlusion-Aware Spatial Masked Attention:

Standard cross-attention allows any image token to attend to any text token. In our processor, the raw attention matrix \(\mathbf{S} = \frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}\) is modulated with a structural attention bias \(\mathbf{M}_{\text{attn}}\):

$$ \mathbf{A}{\text{attn}} = \text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}} + \mathbf{M}{\text{attn}}\right) $$

For image patch \(j\) and text token \(t \in \text{call_ids}_i\):

$$ \mathbf{M}_{\text{attn}}(j,t) = \begin{cases} 0, & \text{if } j \in \mathcal{M}_i \text{ (inside object } i \text{'s visible 3D mask)} \ -\infty, & \text{otherwise} \end{cases} $$

This enforces zero attention leakage outside each object's exact 3D projected boundaries, enabling faithful synthesis even under heavy occlusions and overlaps.

3. End-to-End LoRA Training Pipeline & Flow Matching

We train the model using flow matching on rectified flow trajectories with mixed-precision acceleration.

  • Flow Matching Objective:

Given latent data \(\mathbf{x}_0 \sim p_0\) (encoded target image via VAE, scaled by \(\sigma = 0.1159\)) and standard Gaussian noise \(\mathbf{x}_1 \sim \mathcal{N}(0,\mathbf{I})\), we sample continuous timesteps \(t \in [0,1]\) using logit-normal density sampling. The noisy state is linearly interpolated along the flow line:

$$ \mathbf{x}_t = (1-\sigma_t)\mathbf{x}_0 + \sigma_t\mathbf{x}_1 $$

The objective trains the transformer velocity predictor \(\mathbf{v}_{\theta}\) to regress the true vector field \((\mathbf{x}_1-\mathbf{x}_0)\):

$$ \mathcal{L}{\text{Flow}} = \mathbb{E}{t,\mathbf{x}0,\mathbf{x}_1}\left[ w(t)\cdot \left| \mathbf{v}{\theta}\left( \mathbf{x}t,t,\mathbf{I}{\text{layout}},{\mathcal{M}i}, \mathbf{c}{\text{text}},\mathbf{c}_{\text{pooled}} \right) -(\mathbf{x}_1-\mathbf{x}_0) \right|_2^2 \right] $$

where \(w(t)\) is the noise scheduler loss weighting.

  • Two-Stage Training Curriculum:

    1. Stage 1 (Geometric Alignment, 50 Epochs): Trains the multi-stream LoRA layers on full synthetic compositions to establish 3D bounding box grounding, perspective convergence, and occlusion boundaries across varied camera frustums.
    2. Stage 2 (Photorealistic Refinement, 5,000 Steps): Fine-tunes with hard negative filtering (excluding non-realistic renders) and refined text captions to eliminate residual synthetic artifacts and improve photorealism.

  • Exact Training Hyperparameters:

    • Base Weights: FLUX.1-dev (frozen VAE, frozen T5-XXL, and frozen CLIP-L).
    • LoRA Hyperparameters: Rank \(r = 128\), scaling factor \(\alpha = 128\) (yielding \(\alpha/r = 1.0\)), applied across all 19 double-stream and 38 single-stream attention layers (Q, K, V, O).
    • Precision: bfloat16 mixed precision with fp32 master LoRA parameters and gradient checkpointing enabled.
    • Optimizer: AdamW with \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\), and weight decay \(10^{-4}\).
    • Learning Rate: Peak learning rate \(\eta = 1 \times 10^{-4}\) with a 500-step warmup and constant schedule.
    • Guidance & Resolution: Guidance scale \(s = 1.0\) during training (distilled guidance support) with multi-resolution bucketed inputs up to \(1024 \times 1024\) and condition resolution \(512 \times 512\).
    • Max Gradient Norm: Clipped to \(1.0\) for numerical stability.

4. Interactive Web Studio Built with Gradio

To make this sophisticated pipeline accessible to artists, directors, and researchers, we designed a responsive web interface powered by Gradio 5:

  • Decoupled Asynchronous Microservices: The Gradio application communicates with three background headless Blender servers (ports 5001, 5002, 5003) over lightweight local REST endpoints. This decouples instant UI camera feedback from full-scale ray tracing and diffusion sampling.
  • Interactive Spatial Controls: Custom Gradio sliders govern depth \(X\), lateral spread \(Y\), ground elevation \(Z\), metric dimensions (\(w,d,h\)), and azimuth yaw angles \(\theta\), with immediate wireframe updates.
  • Preset Serialization & Project State: Scene configurations are serialized into portable .pkl files with full camera intrinsics, object hierarchies, and sampling hyperparameters, enabling one-click saving, loading, and benchmark reproducibility.

Challenges We Faced

  1. Handling Depth & Occlusions: Standard 2D approaches suffer when objects overlap because pixel overlap creates prompt blending. Solving this required rendering true 3D spatial masks in depth order, so the diffusion model understands which surfaces occlude others.
  2. Synchronizing Headless Graphics with Web UIs: Orchestrating three concurrent Blender render processes alongside a multi-gigabyte diffusion pipeline without memory bottlenecks or socket deadlocks required careful port assignment, cleanup routines, and cached parameter restoration.
  3. Harmonizing Physical Scales: Users naturally struggle to guess metric bounding box dimensions (e.g., what is the metric depth of a vintage armchair versus a golden retriever?). We resolved this by implementing pre-calibrated reference dimension templates and an auto-adjusting harmonization module.
  4. Sub-second UI Feedback: Users need immediate visual confirmation when tweaking depth \(X\), lateral position \(Y\), or elevation \(Z\). We decoupled the lightweight real-time camera preview from the heavy full-resolution diffusion generation pass.

What We Learned

  • Bridging Graphics and Generative Models: 3D coordinates provide invariant representations that text prompts alone cannot accurately describe. Combining ray-tracing primitives with latent diffusion creates greater creative control.
  • Attention Steering in Modern Architectures: Working under the hood of FLUX.1-dev taught us how placeholder token embeddings and spatial feature injections interact across transformer layers.
  • System Architecture for AI Tools: Building real-time interactive AI software requires isolating heavy rendering, GPU model loading, and responsive frontends so the user experience remains responsive.

What's Next for Director3D

  • Full 6-DoF Rotation: Expanding beyond azimuth (\(\theta\)) to full pitch, roll, and yaw rotation matrices for flying or tumbling objects.
  • Interactive In-Browser 3D Viewport: Integrating WebGL/Three.js directly into the UI so users can drag, rotate, and scale boxes directly with gizmos in the viewport.
  • Multi-View Consistent Synthesis: Leveraging the 3D scene graph to generate consistent multi-view video sweeps and parallax camera paths around synthesized scenes.

Built With

Share this project:

Updates

Submission history