Inspiration
Robotics teams often spend significant time rebuilding real environments inside simulators before they can test navigation, planning, or interaction. Meanwhile, modern phones and tablets can already capture RGB-D video, camera poses, and room geometry.
Real2Sim was inspired by a simple question: Can we turn an ordinary mobile room scan into a useful robot-simulation scene automatically?
Our goal was not to reproduce every surface perfectly. We wanted a practical compiler that preserves recognizable objects, produces usable collision geometry, records uncertainty honestly, and exports artifacts that downstream robotics tools can consume.
What it does
Real2Sim converts a synchronized RGB-D room scan into a semantic, collision-ready USD scene.
The pipeline:
- Imports RGB images, metric depth, camera intrinsics, poses, and the reference mesh.
- Converts the capture into the dataset format expected by ConceptGraphs.
- Uses ConceptGraphs to detect objects and fuse observations across multiple camera views.
- Filters transient detections and exports the retained objects into a safe, tool-independent JSON representation.
- Generates a metric, Z-up USD scene containing semantic point-cloud visuals and static collision proxies.
- Plans and validates a kinematic rover route against the reconstructed collision geometry.
- Produces a visual demo and a manifest describing all generated simulation assets.
For our demonstration scan, Real2Sim processed a 596-frame ARKitScenes sequence. ConceptGraphs evaluated 120 frames using stride five and produced 49 retained multi-view objects.
The reconstructed object geometry aligned with the source mesh at a median distance of approximately*1.44 cm, with a 90th-percentile distance of **3.33 cm. About **96.86%* of evaluated points were within 5 cm of the reference mesh.
How we built it
We used an ARKitScenes RGB-D capture as the real-world input. The importer synchronizes color, depth, camera calibration, poses, and mesh data, then converts them into ConceptGraphs' Record3D-compatible structure.
ConceptGraphs performs the perception stage. Its workflow uses open-vocabulary YOLO-World detections, MobileSAM masks, depth projection, and multi-view feature association to build persistent 3D objects rather than treating every 2D detection as a separate item.
We then added a simulation compiler around that output. It:
- Converts the trusted ConceptGraphs object map into strict JSON.
- Preserves semantic labels, point geometry, confidence, and object bounds.
- Creates static axis-aligned collision boxes from each object's 3D extent.
- Authors a metric, Z-up USDA scene.
- Generates manifests for reproducibility and validation.
- Checks the rover's swept footprint against relevant collision boxes every 5 cm along its route.
GPT-5.6 Sol Medium was used through Codex throughout development. It helped inspect unfamiliar datasets and APIs, diagnose coordinate-system and projection errors, implement importers and exporters, design validation tests, generate documentation, and iterate on the demo renderer. The final perception pipeline itself runs locally and does not require an OpenAI API key.
Challenges we ran into
The hardest problem was coordinate-system correctness. RGB-D reconstruction can look plausible while still being mirrored, rotated, or displaced because camera conventions differ between ARKitScenes, Record3D, ConceptGraphs, and USD. We corrected the camera-axis encoding and validated the resulting object map directly against the source mesh.
Detection quality was another challenge. Single-frame detections can overlap, disappear, or produce fragmented objects. Instead of writing a rushed scene-graph system, we used ConceptGraphs' established multi-view association workflow and filtered out objects that lacked sufficient repeated observations.
Our first rover route also looked visually safe but started inside a secondary component labeled as a bed. We fixed this by validating the rover footprint continuously against vertically relevant object bounds instead of trusting a top-down screenshot.
Finally, the installed Isaac Sim environment crashed inside its RTX viewport initialization before opening our scene. We did not present that as successful Isaac execution. Instead, we validated the USD with Pixar USD, created a deterministic standalone renderer, and clearly described the rover as a collision-checked kinematic demonstration rather than autonomous navigation or PhysX simulation.
Accomplishments that we're proud of
We are proud that Real2Sim produces inspectable, reproducible artifacts rather than only an attractive visualization.
The MVP includes:
- A complete real RGB-D-to-USD pipeline.
- A 49-object semantic reconstruction aligned with the source mesh.
- Static collision proxies for every retained object.
- Strict JSON schemas and simulation manifests.
- A validated 1.68-meter rover route with a 0.16-meter footprint.
- Automated tests for importing, conversion, export, route validation, and video generation.
- A one-command judge quickstart.
- Honest documentation of what is implemented and what remains future work.
We also kept the repository independent of private API keys and separated third-party scan artifacts from the MIT-licensed source code.
What we learned
We learned that semantic reconstruction is only one part of real-to-simulation conversion. A detected object is not automatically simulation-ready. Simulation also requires coordinate consistency, scale, collision geometry, mobility classification, physical properties, provenance, and validation.
We also learned that multi-view evidence is much more reliable than reasoning from a small set of visually interesting frames. Processing the full sequence with a controlled stride produced better coverage and more defensible object persistence.
Most importantly, we learned to distinguish between visually convincing output and verified behavior. Geometry alignment, collision checks, schemas, and reproducible manifests are more valuable than claiming simulator capabilities that were not successfully tested.
What's next for Real2Sim: Scan-to-Simulation Compiler
The current MVP intentionally uses static axis-aligned collision boxes. The next version will classify objects as static, movable, or articulated and propose physical properties such as mass, density, friction, and restitution with confidence and provenance.
We also plan to add:
- Oriented bounding boxes, planes, convex hulls, and mesh colliders.
- Support-surface and containment relationships.
- A review interface for correcting uncertain labels and geometry.
- Navigation-map generation and dynamic simulator validation.
- Articulated replacement assets for doors, drawers, and furniture.
- A richer scene graph connecting objects, regions, spatial relationships, and physical constraints.
- Support for user-provided Polycam and other mobile RGB-D captures.
The long-term goal is a practical compiler that turns everyday scans into editable, physically meaningful environments for robotics development.
Log in or sign up for Devpost to join the conversation.