Planetary Scene Studio

Grok Imagine allowing ideation for base development on Mars surface.
real gaussian splat 3d view of the surface of mars, created from stitching photos from perserverance rover

IMPORTANT: ACCESS CODE TO TRY IT OUT!!!

Access code: 19E6-DAD1-B0D4 → Mission control (demo), sees Mars and Moon

Inspiration

NASA rovers collect extraordinary data, but teams still analyze much of it through disconnected images, maps, and tools. We wanted to turn that data into a shared place where scientists, engineers, and designers can explore terrain and iterate on mission plans together.

What it does

Planetary Scene Studio converts real NASA rover imagery and orbital data into a collaborative 3D planning environment.

Our Mars scene contains 972,047 trained Gaussians, reconstructing the area around Perseverance’s home base. 97.4% of those Gaussians carry a machine-learned terrain classification, allowing users to inspect slope, elevation, roughness, and surface type directly within the reconstruction.

Teams can search terrain, place habitats and infrastructure, compare site scores, add shared observations, and plan slope-aware rover routes. SpacetimeDB synchronizes every cursor, pin, rover movement, and base module across all connected users in real time.

How we built it

We reconstructed the Mars site from Perseverance Navcam and Mastcam-Z images using rover camera models, stereo depth, COLMAP-compatible geometry, and Gaussian splatting.

The final compressed scene is 16 MB, renders roughly one million Gaussians at 60 FPS, and is registered onto 1 m/px HiRISE terrain. A model trained on NASA’s AI4Mars dataset adds terrain classifications to the reconstruction.

The viewer uses React, Three.js, and Spark. SpacetimeDB provides the multiplayer state layer. FastAPI and Grok power natural-language commands, voice interaction, and concept renders derived from the current camera view.

Challenges we ran into

Rover cameras are designed for science and navigation, not complete 3D capture. Sparse viewpoints initially produced holes, stretched Gaussians, and artifacts outside the original camera angles.

We corrected stereo alignment to approximately 0.02 px, curated images across multiple rover stops, matched exposure between cameras, and evaluated every training run from novel viewpoints. We also created strict data contracts so the reconstruction, ML layers, browser, and multiplayer backend all used the same coordinates and units.

Accomplishments that we're proud of

Reconstructed a real Perseverance site as 972,047 interactive Gaussians
Classified 97.4% of the reconstruction with ML-derived terrain labels
Achieved approximately 60 FPS with a 16 MB scene bundle
Synchronized users, designs, observations, and rover drives through SpacetimeDB
Built natural-language, voice, and image-generation workflows with Grok
Added a Moon environment using LOLA terrain and LRO imagery
Built the complete system in one night with Cursor

What we learned

Immersion is only valuable when users can trust the underlying data. Every scientific layer needs a source, resolution, and clear uncertainty label.

We also learned that Gaussian splatting can be more than a visualization technique. When each Gaussian carries geometry and machine-learned information, the reconstruction becomes a spatial dataset that teams can query and design against.

What's next for Planetary Scene Studio

Next, we want to support additional rover sites, richer scientific measurements, and persistent planning rooms for research and education.

Our larger goal is to turn planetary mission data into shared, interactive environments where teams can evaluate terrain, explore scientific targets, and design humanity’s next foothold beyond Earth.