Astra: An Engine-Neutral World Compiler for Embodied Intelligence

Inspiration

World generation has largely been optimized for a human observer: if an environment appears coherent from a camera, it is treated as complete. Embodied agents impose a more difficult criterion. They encounter the latent structure behind the image—scale, free space, contact, support, articulation, dynamics, and the causal consequences of action.

The relevant unit of world generation is not an image or a mesh, but an executable hypothesis about how a world behaves.

For a world (\mathcal{W}), policy (\pi), and embodiment (b), Astra is concerned with the transition system induced by their interaction:

$$ \tau_{t+1} \sim P_{\mathcal{W}}!\left(\tau_{t+1}\mid\tau_t,\pi,b\right) $$

A useful world must preserve the task-relevant structure of this transition system—not merely its rendered appearance.

What it does

Astra is our world compiler for spatial intelligence. Its live path already converts generated spatial content into canonical, simulator-ready worlds containing visual appearance, metric geometry, collision, articulated assets, semantic identities, affordances, layouts, tasks, and validation evidence.

The system maintains a dual representation:

\mathcal{W}

\mathcal{W}{\mathrm{visual}} \oplus \mathcal{W}{\mathrm{physical}} $$

The visual layer uses splats and render geometry to preserve the character and spatial presence of a generated environment. The physical layer carries the structure required for policy execution: collision geometry, support constraints, joint models, object state, spawn regions, affordance predicates, and runtime packages.

Astra is not coupled to one simulator. Its canonical world contract is compiled through engine adapters, allowing the same environment to support policies across simulation stacks. The current path produces packages for systems including MuJoCo, Genesis, Isaac, Drake, Unity, and Unreal, while retaining an engine-neutral source of truth for additional targets.

Astra also generates controlled world families rather than isolated scenes. Layouts, asset variants, and digital cousins can alter appearance, morphology, or spatial arrangement while preserving task-relevant invariants:

\mathcal{I}_{T}(\mathcal{W}')

\mathcal{I}_{T}(\mathcal{W}) $$

where (\mathcal{I}_{T}) denotes the affordances and constraints required by a task family (T). One world can therefore become a structured distribution for policy training, robustness testing, and evaluation.

We are now developing Astra Real2Sim, a research path within the same system. It reconstructs observed environments from video and enters Astra at the world-representation boundary. Real2Sim is not a separate product and is not yet the live generation path; it extends Astra’s compiler from generated worlds to evidence-grounded reconstructions of real environments.

How we built it

We designed Astra as a sequence of constrained compilation stages rather than a monolithic generative pass. The system resolves scene structure, places and settles objects, constructs collision representations, preserves articulation, generates task-compatible variants and cousins, builds affordance graphs, exports simulator packages, and validates the resulting world at each irreversible boundary.

The compiler must preserve a set of invariants across every target engine (e):

$$ \Phi_e!\left(C_e(\mathcal{W})\right) \simeq \Phi(\mathcal{W}), \qquad e \in \mathcal{E} $$

Here, (C_e) is an engine adapter and (\Phi) denotes the world properties relevant to policy execution: scale, topology, collision, kinematics, semantics, and task predicates. Exporting a file is therefore not sufficient; the compiled world must reload and remain behaviorally meaningful.

Real2Sim adds a complementary evidence pipeline: camera and metric recovery, multi-view depth, persistent object identity, support-surface inference, bounded image completion, separable asset reconstruction, physics-based settling, and promotion into Astra’s canonical representation.

We used Codex with GPT-5.6 Sol as a research instrument across this process. Sol helped us triangulate between code, geometry, simulation behavior, visual evidence, and evaluation traces; generate competing explanations for failures; and collapse uncertainty through implementation and experiment. Its role was not to replace measurement, but to sustain coherent search across an underdetermined problem space in which no single representation contains the answer.

Challenges we ran into

The hardest failures occurred at representation boundaries. A visually plausible object may have invalid mass distribution. A correct mesh may have unusable collision. A stable scene may lose articulation during export. A diverse variant may no longer satisfy the task that made its source world useful.

These errors often remain invisible until a policy acts. We therefore had to treat generation as proposal, compilation as constraint satisfaction, and simulation as falsification.

Real2Sim introduces an additional epistemic problem: distinguishing observation from inference. Occlusion and incomplete coverage make generative completion necessary, but unconstrained completion can corrupt the evidence required for physical fidelity. Astra preserves measured evidence as authoritative and limits generative completion to explicitly bounded regions.

Accomplishments that we're proud of

We built the layer that is usually omitted between world generation and embodied learning: a complete compilation path from spatial content to executable policy environments.

A live sim-ready world compiler. Astra already converts generated worlds into structured physical environments rather than stopping at a render, splat, or mesh.
An engine-neutral policy substrate. One canonical world can be compiled across major robotics, simulation, and interactive engines without making any individual engine the semantic source of truth.
Coupled visual and physical worlds. Splats preserve appearance while meshes, colliders, articulated assets, and stateful scene structure provide an independent substrate for interaction.
Constraint-aware scene synthesis. Placement and layout are governed by support, accessibility, collision, and task feasibility rather than visual plausibility alone.
Mechanically meaningful articulation. Joints, links, limits, collision geometry, and reset states survive into simulator-facing packages and can be exercised by embodied agents.
Invariant-preserving variation. Astra produces layouts, variants, and digital cousins that expand the training distribution while retaining the affordances required by the original task.
Multi-embodiment execution. The same world representation supports manipulation and locomotion, demonstrated with a Franka arm and a quadruped operating against shared scene structure.
Research-grade acceptance. Worlds carry provenance and validation artifacts for geometry, collision, articulation, variants, and runtime behavior; readiness is a claim to be tested, not a label attached to an export.
Room-scale world construction. Astra expands sim-ready generation beyond bounded object arrangements into continuous environments with multiple support surfaces, navigable space, heterogeneous assets, and multiple classes of embodied behavior.

The resulting object is not simply a scene. It is a parameterized family of executable worlds:

$$ \left{\mathcal{W}{\theta} : \theta \in \Theta{\mathrm{valid}}\right}, \qquad

\Theta_{\mathrm{valid}}

\left{\theta : \mathcal{C}(\mathcal{W}_{\theta}) = 1\right} $$

where (\mathcal{C}) represents the geometric, physical, semantic, and task constraints required for admission.

What we learned

The most important lesson is that simulation readiness is not a property of an asset. It is a property of a complete system of relationships.

World quality is therefore irreducibly multi-objective:

Q(\mathcal{W})

f!\left( Q_{\mathrm{appearance}}, Q_{\mathrm{geometry}}, Q_{\mathrm{kinematics}}, Q_{\mathrm{physics}}, Q_{\mathrm{tasks}}, Q_{\mathrm{transfer}} \right) $$

No single metric can certify the whole world. Visual fidelity, physical validity, runtime portability, and policy relevance must remain separately inspectable and jointly constrained.

We also learned that generative models become more useful as their authority becomes more explicit. They are powerful mechanisms for proposing structure and exploring hypotheses, but evidence, constraints, and executable evaluation must decide what enters the world.

What's next for it

Our immediate focus is completing Astra Real2Sim and evaluating the unified system across generated and reconstructed room-scale environments. We are expanding cross-engine consistency tests, policy-transfer studies, uncertainty representations, articulated-object reconstruction, physical-parameter estimation, and automatic task generation.

The long-term goal is to make worlds into scientific instruments for embodied intelligence: portable enough to run anywhere, structured enough to support intervention, and faithful enough that behavior in simulation becomes informative about behavior in reality.