4D-SynthForge

Vibe Coding for Robots. 4D-SynthForge turns one real-world video into infinite, physics-accurate synthetic training datasets using Gemini and Isaac Sim.

Comment

Vibe Coding for Robots: Turning Reality into Infinite Data.

Training robots today is stuck in the dark ages. Engineers spend hundreds of hours manually coding URDF files, guessing friction coefficients, and tweaking physics parameters. It’s rigid, slow, and completely devoid of "flow."

4D-SynthForge changes the paradigm.

We built the first tool that lets you "Vibe Code" a physical simulation. You don't write code; you direct it. You simply show the AI a video of what you want—a ball bouncing, a robot arm moving, a drone flying—and 4D-SynthForge handles the rest.

The Workflow: Reality to infinite Simulation

  1. Capture the Vibe 📹 (Video Input)

    • You upload a simple video of a physical event.
    • Gemini 3.0 doesn't just see pixels; it extracts the physical priors. It estimates mass, velocity, restitution, and material properties based on visual intuition.

  2. Generate the Twin 🔧 (Code Generation)

    • The system automatically writes the Python USD/PhysX code for NVIDIA Isaac Sim.
    • A fully rigged, physics-accurate 3D scene appears instantly. No manual coding required.

  3. Scale the Vibe 🎲 (Domain Randomization)

    • One simulation isn't enough for training. So we multiply it.
    • Using Omniverse Replicator, we create 100+ parallel universes. We vary lighting, textures, friction, and gravity—all while keeping the original "vibe" intact.

  4. Train the Brain 🤖 (Synthetic Data)

    • We export massive, labelled datasets ready for LeRobot, Rerun.io, or any other robotics learning framework.

Why "Vibe Coding"?

Because precise physics measurement from video is impossible, but capturing the intent and approximate dynamics is not. We use AI intuition to get the "vibe" right, and then use Domain Randomization to robustly cover the reality gap. This is the future of synthetic data generation: fast, intuitive, and infinite.

🛠️ Built With

  • Gemini 3.0 Pro/Flash (Multimodal Physics Extraction & Code Gen)
  • NVIDIA Isaac Sim (High-Fidelity Physics Simulation)
  • Universal Scene Description (USD) (Open 3D Standard)
  • PhysX 5 (Real-time Physics Engine)
  • Python 3.10 (Core Logic)

Built With

Share this project:

Updates