GK&KG LA45A splat-foley -

splat-fole An automatic sound-design pipeline that turns any 3D content meshes, scans, Gaussian-splat captures, a Marble-generated splat world, or a TRIPO-generated asset from a text prompt

Comment

Inspiration

Every 3D asset ships silent. With Marble conjuring whole splat worlds and TRIPO conjuring objects from a sentence, 3D content is becoming effectively free — sound design is not. A sound designer still looks at each object, reasons about its materials, sources every impact, drag and creak by hand, and edits each one to a clean hit or a seamless loop. Our observation: most of the information a sound designer uses is already in the geometry. A vision model can see it's a slatted wooden crate; a reasoning model knows a crate affords kicks, drags, creaks and wind; a text-to-audio model can render each of those; and an audio model can listen to the result and check it. So we chained them.

What it does

splat-foley takes any 3D input — meshes, photogrammetry scans, Gaussian-splat captures, a whole Marble world, or just a text prompt (Tripo3D for objects, World Labs Marble for worlds) — and emits a complete, verified sound world:

  1. Ingest & render — robust normalization for real captures (COLMAP Y-flip, floater culling, dense-core re-normalization), six orbit views via a pure-numpy rasterizer or true 3DGS rendering with gsplat.
  2. See — a vision LLM (Qwen3-VL-8B) writes an editable property document: objects, materials, size, condition. SAM cuts out parts (hinges, lids, straps, clappers), each labeled with part_of provenance.
  3. Plan — an LLM turns the properties into 8–25 physically plausible effects per asset plus loopable ambience beds. Every prompt is an acoustic description of the sound, never event narration.
  4. Generate & verify — AudioLDM2 renders every effect; LAION-CLAP scores each wav against its own prompt (accept ≥ 0.25, seeded retries, N takes kept).
  5. Refine — the pipeline listens. RMS-envelope analysis finds exact event timings; Qwen2-Audio writes a timed transcript of what it hears. Multi-hit impacts get cut to the single best hit; loopables get equal-power crossfade seamless loops; every edit re-verified, with automatic rollback.
  6. Package — one zip per asset: model + textures + property JSON + every audio take (refined, variations, raw) + a manifest with scores and exact timings. Drop the zip back on the web GUI and the asset re-imports.

Everything is editable live in a web GUI, and every stage re-runs from plain files on disk.

How we built it

Local-first, by design. All AI stages run on our own hardware — an RTX 4090 workstation (SAM ViT-H, gsplat CUDA rendering) plus an 8-GPU studio server (Qwen3-VL-8B on vLLM, AudioLDM2, Qwen2-Audio + CLAP, and the TTS/music stack that voiced the trailer). The biggest model in the chain is 8B; the whole sound pipeline fits in ~57 GB of resident VRAM across cards no larger than 24–48 GB. Only the two generative-3D conjure calls (Tripo3D, Marble) leave the LAN — their results arrive as local files.

The architecture is deliberately boring: nine stages communicating exclusively through hand-editable JSON/wav/png on disk. Any stage re-runs after you edit the previous stage's output; a service failing mid-run never aborts anything. A cloud twin runs as a Hugging Face Space with in-process models — a full generation verified in 68 seconds on the free CPU tier. Even the trailer was produced by the same machinery: IndexTTS-2 narration, ACE-Step score (vocal-gated by Qwen2-Audio), and splat-foley's own refined effects as the accents.

Challenges we ran into

  • Text-to-audio prompt style makes or breaks everything. "Box gets kicked" produces mush; "a single hollow wooden thud, boot kicking an empty crate" works. Rewriting prompts as concrete acoustic descriptions took the crate from 5 weak effects to 20/20 accepted.
  • CLAP is biased against short clips. Our first refine pass rolled back every correct trim, because a clean 0.4 s hit scores worse than 3 s of mush. Fix: verify on comparable signals — one-shots tiled back to ≥4 s, loops scored played twice (which also proves the seam).
  • Real captures fight you. Y-down orientations, floaters that shrink the scene to a speck, sky-shell gaussians smearing every view, outdoor scenes that only look right from inside. Each got a dedicated fix.
  • Soft materials are the honest failure mode. Wicker and felt sit at 25–41% acceptance — AudioLDM2 and CLAP both struggle with faint sounds. Hard resonant materials are essentially solved (treasure chest: 19/19).

Accomplishments that we're proud of

  • 16 assets, 254 verified effects, 72% first-pass CLAP acceptance — including real scans, real splat captures, a Marble world and TRIPO objects, all through one pipeline.
  • A refine stage that audibly improves its own output: a 4-hit plank impact cut from 3.0 s to a single 0.43 s hit while its comparable CLAP score rose from 0.457 to 0.568.
  • The trailer proves the product: its entire soundtrack — narration, score, every accent — is generated audio, and the sound effects in it are splat-foley's own refined output.
  • Fully local: the whole chain runs on hardware enthusiasts own; nothing bigger than 8B anywhere.
  • A self-contained asset-package format that round-trips through the GUI, with provenance down to per-event millisecond timings and the machine's own listening transcript.

What we learned

Verification is the product. Generation is easy; knowing which generations are right — and rolling back your own edits when the numbers say so — is what makes an automatic pipeline trustworthy. Comparability matters more than the metric: the same CLAP that misleads on raw short clips becomes reliable once you score like against like. And an 8B vision model, a 1.1B audio model and a disciplined file contract beat a giant monolith you can't inspect.

What's next for GK&KG LA45A splat-foley

Soft-material prompt strategies to lift wicker and felt; spatial audio placement derived from the SAM masks; per-engine export layouts (Unity/Unreal); and wiring the sound pass to run automatically behind every Marble world and TRIPO asset the moment it's conjured — so nothing ever ships silent again.

Built With

Share this project:

Updates