Inspiration
On Mars, a message to Earth and back takes about 25 minutes. A base can't wait for headquarters to approve every decision, so it needs a crew that can run itself. The same problem exists on Earth: remote stations, ships, and small businesses that run with no back office.
What it does
You can put on a Google Cardboard headset and become the commander of a base in Jezero Crater, built from real NASA terrain. In this simulation, five AI agents run the base: Engineer (power and machines), Life (air and water), Science (the lab), Supply (cargo, bills, payments), and Rules (oversight). A dust storm cuts the power, a supplier sends a quote at triple the price to a new bank account, Rules blocks it as a scam, and the crew brings you one decision: what do we give up? Your choice changes the base, visibly. Afterward, the base fades into a real 360° Perseverance panorama.
There is also a Mac spectator view mirrors the judge's head live with subtitles, so everyone at the table sees what the commander sees.
How we built it
- VR app: native Swift + SceneKit + the Google Cardboard SDK on iPhone, stereo at 60 fps; a macOS spectator app shares the scene.
- Agent engine (TypeScript): each agent is an LLM with tools limited to what its role is allowed to do. Agents only propose; deterministic code checks every proposal against an editable policy (three-way match, approval limits, segregation of duties, duplicate and bank-change checks, statistical price checks, cash runway) before anything happens. A double-entry ledger with a full audit trail.
- Three model layers: cloud model first, a local model (Ollama) offline, and deterministic rules as the floor. A safe-mode switch drops to rules instantly.
- Real data: USGS/NASA Jezero terrain, the Mars24 sun model, the real Earth–Mars light time, solar power calibrated on Opportunity's 2018 dust storm, a real Perseverance Mastcam-Z stereo pair viewed in true 3D, and the Van Zyl Overlook panorama. Prices and suppliers are simulated and labeled.
Proof it isn't hardcoded
- The situation (storm, short delivery, scam quote) is scripted; the agents' decisions and lines are generated live and validated against the engine's facts.
- 30 seeds with live agents: every promised outcome held 30/30; 4% of decisions fell back to rules; median decision 2.8 s.
- 100 held-out seeds on the rules engine: 0 payments to fraudsters, 0 ledger integrity failures, 100% recall on blatant errors, 35% on subtle +3–7% overpricing (shown openly).
- Judges can type any crisis on the laptop and watch the crew handle it.
- Multiple rounds of adversarial review; 95 tests plus blocked-exploit repros.
Challenges
Making agents genuinely autonomous while guaranteeing they can never move money unsafely is an extremely hard challenge. Also, keeping live AI lines short, readable, and on-story. We also had technical challenges, such as rendering real terrain on a phone at 60 fps and making a VR demo watchable for a whole table of judges.
What we learned
The interesting part of autonomy is knowing when not to act alone. The rules make the dangerous moments reach a human every time.
What's next
The engine never mentions Mars, so the same crew can run procure-to-pay for remote stations and small businesses with no finance team.
Built With
- gdal
- google-cardboard
- ios
- macos
- node.js
- ollama
- openai
- scenekit
- swift
- typescript
- websocket
- xcode
Log in or sign up for Devpost to join the conversation.