-
-
editor and AI
Steinmetz — select a rough blockout in our Rust/Vulkan level editor, type a prompt, and a validated, textured AI-generated mesh replaces it in place — three swappable model lanes (local TRELLIS, local Hunyuan3D, Tripo3D cloud).
STORY — Inspiration
Level blockouts are fast. Detail is slow. The octree-and-texture workflow that makes a Cube-lineage editor pleasant for roughing out a space makes ornate geometry — carved doorways, cluttered crates, weathered walls — a grind. AI 3D generation should close that gap, but today the models live in notebooks and web UIs: nothing places the result, nothing validates it, and every model demands its own integration.
Steinmetz is German for stonemason — the craftsman who turns a rough-hewn block into carved ornament. We chose the name deliberately: the artist keeps blocking out in stone; the mason carves the detail in place.
(And it's the sibling of splat-foley: TRIPO and Marble conjure the 3D, one teammate makes it sound, the other makes it carve. Nothing ships silent — and nothing stays a blockout.)
STORY — What it does
The loop is one dialog. In the editor: select geometry, open File → AI Detail…, type a prompt, pick a lane, submit. Seconds later a validated, textured mesh stands exactly in the selected volume, the blockout it replaced carved away (one undo restores it). Hit Accept and it persists with the map — reloaded every time that map opens.
Behind the dialog sit three genuinely different, swappable provider lanes:
- TRELLIS variant (local) — conditions on the source mesh AND your text prompt. The only lane that consumes the prompt today, and we say so.
- Hunyuan3D-2mv Turbo (local) — shape-conditioned from orthographic views of your selection, rendered by the editor's own pure-CPU rasterizer, behind a hardened ComfyUI gateway.
- Tripo3D (cloud) — the same view renders, egress-gated: only those PNGs ever leave the machine. The source mesh and the prompt never do, and every cloud job requires an explicit per-job opt-in.
We're honest about adherence: the multiview lanes ignore the text prompt (their APIs have no text input), so the service records an explicit warning on every such job and the UI states each lane's real conditioning mode. Every result is envelope-validated, triangle-budgeted, content-addressed, and provenance-stamped before it's allowed near your map.
STORY — How we built it
The editor is Cube Bauhaus, our native Rust/Vulkan editor in the Cube 2 octree lineage. The AI side is MeshDetail, a Python/FastAPI control plane with content-addressed artifact storage and SQLite job records. The core never imports torch, TRELLIS, ComfyUI nodes, or Blender — every model lives in its own pinned environment behind a versioned detail-provider/v1 contract.
Cube Bauhaus editor (Rust/Vulkan, Cube2 octree lineage)
| select -> clip to selection AABB -> centered [-0.5,+0.5] unit-box GLB
| + column-major world_from_normalized placement matrix
| + deterministic CPU-rendered orthographic condition views (PNG)
v
MeshDetail control plane (local, loopback-only)
|- content-addressed assets (SHA-256), detail-jobs API, poll/cancel
|- GLB/envelope/triangle validation, provenance, cache
|- resource manager: exclusive CUDA lease + VRAM re-probe + reclaim
`- provider registry
|- trellis_variant isolated WSL subprocess mesh+prompt
|- comfyui_local comfy-detail-gateway/v1 render_conditioned
|- tripo_remote egress-gated HTTPS render_conditioned
`- passthrough no-model baseline proving the whole contract
v
validated detailed.glb + placement transform back into the editor world
Notable pieces: the ComfyUI gateway speaks its own versioned protocol with idempotency keys and verified mid-generation cancellation, and refuses workflows that would silently ignore the selection; a headless Blender conditioner is the server-side fallback when a caller sends no view renders; WSL mirrored networking lets the WSL control plane and the Windows-native gateway share strict loopback-only binding; and one exclusive GPU lease governs the RTX 3090 across heterogeneous model runtimes, with automatic VRAM reclaim between them. Checkpoints, workflow JSON, and commits are hash-pinned, and every result carries that provenance.
STORY — Challenges we ran into
- One 24 GB GPU, two hungry residents. TRELLIS peaks at 16.5 GB while a warm ComfyUI holds its own memory. The lease manager re-probes free VRAM and asks the gateway to unload warm models before failing a job — automatic reclaim instead of manual restarts.
- Coordinate contracts lie. Hunyuan3D outputs roughly [-1, 1] geometry against our [-0.5, +0.5] unit-box contract. A deterministic 0.5 rescale keeps validation honest instead of loosening the envelope. (Tripo, we measured, already speaks unit-box — so no rescale there, learned the half-size way.)
- Multiview APIs silently ignore prompts. Rather than pretend, we made adherence modes explicit, record a warning on every prompt-less job, and surface the truth in the pipeline picker UI.
- Windows/WSL loopback split. WSL2's default NAT can't reach Windows 127.0.0.1; mirrored networking keeps both services strictly loopback-only under one resource manager.
- A mouse that snapped and a paste that dropped cubes. The editor's cursor- warp state machine desynced under Windows event coalescing — replaced with raw input. And our Cube2 paste port iterated the destination selection's 1x1x1 size instead of the clipboard's, silently dropping every cube but one — a bug we traced back to semantics the original engine had and our port lost.
STORY — Accomplishments that we're proud of
Every number is from verified runs: Hunyuan3D local generations in ~21 s (~26 s full end-to-end at a 6.45 GB VRAM peak); TRELLIS in 58.5 s; the cloud lane in ~117 s at 499,352 triangles against a 500,000 face limit we learned to forward. A 719k-triangle medieval tower reconstructed from four editor-rendered views. 100+ automated tests across Rust and Python, including a 35/35 parse of every installed community map. And the full loop works live: select, generate, validate — the result replaces the blockout at its exact location and persists with the map.
STORY — What we learned
The durable value isn't any single model — it's the validated, provenance- carrying loop between the artist's selection and the placed result. A passthrough no-model provider that proves the whole contract was the best testing decision we made. And measuring beats believing: PyTorch's allocator peak understates real VRAM by 6 GB, and the first cloud run came back at 2.8x the triangle budget until we forwarded the constraint.
STORY — What's next for GK&KG LA45A Steinmetz
Richer conditioning views, in-editor editing of accepted props, more provider lanes behind the same contract — and wiring splat-foley behind every accepted prop so each carved object arrives with its own verified sound world. Models will keep changing; the contract is built so they can.
Log in or sign up for Devpost to join the conversation.