Inspiration

On Mars, a message to Earth and back takes about 25 minutes. A base can't wait for headquarters to approve every decision, so it needs a crew that can run itself. The same problem exists on Earth: remote stations, ships, and small businesses that run with no back office.

What it does

You can put on a Google Cardboard headset and become the commander of a base in Jezero Crater, built from real NASA terrain. In this simulation, five AI agents run the base: Engineer (power and machines), Life (air and water), Science (the lab), Supply (cargo, bills, payments), and Rules (oversight). A dust storm cuts the power, a supplier sends a quote at triple the price to a new bank account, Rules blocks it as a scam, and the crew brings you one decision: what do we give up? Your choice changes the base, visibly. Afterward, the base fades into a real 360° Perseverance panorama.

There is also a Mac spectator view mirrors the judge's head live with subtitles, so everyone at the table sees what the commander sees.

How we built it

  • VR app: native Swift + SceneKit + the Google Cardboard SDK on iPhone, stereo at 60 fps; a macOS spectator app shares the scene.
  • Agent engine (TypeScript): each agent is an LLM with tools limited to what its role is allowed to do. Agents only propose; deterministic code checks every proposal against an editable policy (three-way match, approval limits, segregation of duties, duplicate and bank-change checks, statistical price checks, cash runway) before anything happens. A double-entry ledger with a full audit trail.
  • Three model layers: cloud model first, a local model (Ollama) offline, and deterministic rules as the floor. A safe-mode switch drops to rules instantly.
  • Real data: USGS/NASA Jezero terrain, the Mars24 sun model, the real Earth–Mars light time, solar power calibrated on Opportunity's 2018 dust storm, a real Perseverance Mastcam-Z stereo pair viewed in true 3D, and the Van Zyl Overlook panorama. Prices and suppliers are simulated and labeled.

Proof it isn't hardcoded

  • The situation (storm, short delivery, scam quote) is scripted; the agents' decisions and lines are generated live and validated against the engine's facts.
  • 30 seeds with live agents: every promised outcome held 30/30; 4% of decisions fell back to rules; median decision 2.8 s.
  • 100 held-out seeds on the rules engine: 0 payments to fraudsters, 0 ledger integrity failures, 100% recall on blatant errors, 35% on subtle +3–7% overpricing (shown openly).
  • Judges can type any crisis on the laptop and watch the crew handle it.
  • Multiple rounds of adversarial review; 95 tests plus blocked-exploit repros.

Challenges

Making agents genuinely autonomous while guaranteeing they can never move money unsafely is an extremely hard challenge. Also, keeping live AI lines short, readable, and on-story. We also had technical challenges, such as rendering real terrain on a phone at 60 fps and making a VR demo watchable for a whole table of judges.

What we learned

The interesting part of autonomy is knowing when not to act alone. The rules make the dangerous moments reach a human every time.

What's next

The engine never mentions Mars, so the same crew can run procure-to-pay for remote stations and small businesses with no finance team.

Built With

Share this project:

Updates

Submission history