Exulanica

The Open World Engine for AI

Comment 1

Exulanica is the open world engine for AI: a persistent 3D world runtime where open models, external agents, and humans operate in the same verifiable world under the same rules.

Today, 3D world generation and AI agents are built in separate silos:

  • Video world models (like DeepMind Genie and NVIDIA Cosmos) generate photorealistic video frames from prompts or control inputs, and models like Cosmos Transfer are specifically designed to render over depth and segmentation buffers from a 3D engine. On their own, pixel generators are not meant to be the persistent database underneath where objects keep permanent IDs, where rules are checked on every turn, or where dozens of language models interact over hours.
  • Traditional game engines (like Unity and Roblox) track 3D physics and state for human players, but were not built to host language models: they lack per-decision spend caps, bounded perception prompts, full decision logs, and deterministic replay of model turns without re-calling the API.
  • Standard AI benchmarks test models on isolated text or web tasks, with no shared space, no other minds around them, and no state that carries forward from one hour to the next.

Exulanica is the runtime layer between them. The engine maintains the authoritative 3D world state (World <-> Data, where every building, street, vehicle, traffic signal, creature, and person has a permanent ID and exact coordinates) and uses a pluggable decision-role registry so any entity or system in the world that makes choices can be driven by a model or external agent. Models never mutate the world directly: they propose an action from the legal options available, the engine validates that proposal against the world's rules before applying it, and every step is recorded so any run can be inspected, branched, or replayed.

Who It Is For

  • Students and teachers first. "AI literacy is different from AI tool use" (OECD, PISA 2029). In Exulanica, a class can open any decision a model made and see what it perceived, the choices it had, which one it picked, and what it cost, swap the model to watch what changes, and save the entire world and its decision history straight to their own GitHub repository.
  • Open-model and agent builders. Watch NVIDIA Nemotron, Qwen, DeepSeek, or your own external agent act alongside other models under real rules, then export the engine-checked runs as SFT and DPO training files for NVIDIA NeMo Gym.
  • Next, games and physical-AI simulations. Persistent worlds where not just characters, but creatures, traffic signals, autonomous vehicles, and world systems run on open models, with every move checked by the engine.

Three Measurable Problems Driving This Project

  1. Nearly 3 Million Open Models Go Unused: Hugging Face hosts 2.96 million public model repositories, yet 85.6% have fewer than 200 lifetime downloads, and just 1.5% of repositories account for 99.2% of all downloads (Hugging Face, State of Open Models: Summer 2026). A static benchmark table or a single chat prompt cannot show how an open model actually behaves over time when it has to share a world with other models and people.

  2. The AI Trust and Literacy Gap: 50% of U.S. adults are more concerned than excited about AI in daily life (up from 37% in 2021), roughly 60% want more control over how AI is used, and 53% are not confident they can tell whether something was made by AI or a person (Pew Research Center, 2025). At the same time, over 80% of U.S. high school and college students and 88% to 96% of European teens use AI tools, while only 6% of teachers say their school has a clear AI policy (Stanford HAI, 2026 AI Index). As the OECD and European Commission AI Literacy Framework for PISA 2029 puts it: "AI literacy is different from AI tool use." People cannot understand or trust AI when decisions happen inside an uninspectable black box.

  3. Distilling Smaller Models Needs an Environment That Checks the Answers: Training a small, fast local model from a larger model usually relies on raw text outputs with nothing verifying whether the large model's choices were actually valid. When models act inside a world engine that checks every move, the engine automatically separates valid actions from invalid proposals and records why a move was blocked, producing clean data for fine-tuning and preference training.

What Exulanica Does

  • Prompt-to-World Generation with Upfront Cost Caps: Describe a place in a sentence. Tavily searches the web for real-world architectural and geographic context, Qwen3 235B condenses those results into reference notes, and NVIDIA Nemotron 3 Super 120B on Nebius Token Factory plans the layout and lighting. Before generation starts, the engine reserves the worst-case cost (Up to $0.26) and records the actual spend when done (around $0.006).
  • 3D World and Live Data Views (World <-> Data): Because the 3D scene is backed by structured records, you can switch at any time between the rendered 3D world and the underlying data view to see semantic point clouds, exact coordinates, object IDs, and live model telemetry above each active agent.
  • Pluggable Decision Roles (Beyond Just Talking NPCs): Instead of hardcoding LLMs into chat characters, Exulanica uses a data-driven decision-role registry where any entity or system with choice points can be handed to a routine, a hosted open model on Nebius Token Factory (Nemotron, Qwen, DeepSeek), a free local model via Ollama, or an external Python/MCP agent. In the live build today, you can assign models to both townspeople and intersection traffic lights, generate custom 3D creatures of any scale with Nemotron (from ankle-high companions to house-sized giants), and plug new roles (such as autonomous vehicles or custom game systems) into the same validation and replay pipeline.
  • Full 5-Stage Decision Breakdown: You can click any turn to inspect all five stages of the decision: 1. World Snapshot -> 2. Context -> 3. Model Inference -> 4. Engine Check -> 5. Outcome. It shows what the model perceived within 8 meters, what it remembered, the shuffled list of legal options it could pick from, which option it chose, the latency and exact sub-cent cost, and the raw JSON prompt and response.
  • Counterfactual Comparisons & Trajectory Export: Because every turn is saved, you can run a world with one model and again with another (such as Nemotron 3 Super 120B, Nano 30B, or Ultra 550B) from the exact same starting state and seeds to compare their behavior side by side, or swap a model mid-run. Validated runs export as supervised fine-tuning (sft.jsonl) and preference (dpo.jsonl) datasets formatted for NVIDIA NeMo Gym, and worlds can be synced directly to a private GitHub repository.

How We Built It (Sponsor Stack & Architecture)

  • Nebius Token Factory (Unified Inference & Cost Control): Every hosted model call routes through a single OpenAI-compatible client pointed at Nebius Token Factory. Having NVIDIA Nemotron, Qwen3, and DeepSeek behind one endpoint and billing API meant we could mix different model families in the same world and enforce strict upfront spend caps per run (which also lets judges and visitors enter as guests with no login or API key).
  • NVIDIA Nemotron Family (Matched by Role):
    • Nemotron 3 Super 120B: Plans 3D world layouts from prompts and web notes (3.6 s median across 12 test prompts, 12/12 valid on the first try), selects world looks and sky/ground settings (1.0 s to 2.8 s median), drafts custom 3D creatures from text (from ankle-high companions to house-sized giants), selects grounded evidence for town Q&A, and runs inhabitants in live worlds.
    • Nemotron 3 Nano 30B: High-frequency, low-cost model for inhabitants and Companion Q&A. In our 60-minute, 8-agent town benchmark (87 decisions), Nano 30B completed the entire hour with zero timeouts, a 1.30 s median latency, and a total cost of $0.0056.
    • Nemotron 3 Ultra 550B: Frontier reasoning option for inhabitants and comparisons; its validated turns can be exported as teacher data for distillation.
    • Nemotron 3.5 Lightning: Powers our external autonomous agent and serves as the fast fallback model.
  • Nebius AI Cloud (Live Production Server & RTX PRO 6000 Serverless GPUs): One Nebius AI Cloud VM hosts the live web stack at exulanica.com (Caddy TLS proxy, PlayCanvas 3D client, FastAPI backend, PostgreSQL database, tile baker, and simulation workers). We also used Nebius Serverless GPU jobs on NVIDIA RTX PRO 6000 Blackwell (96 GB) GPUs to generate 3D town assets and run open video models (Wan2.2) that paint photorealistic views conditioned on the engine's exact depth buffers.
  • NVIDIA NeMo Agent Toolkit & NeMo Gym: We built our external autonomous agent example on the NVIDIA NeMo Agent Toolkit (nvidia-nat), connecting it to the world server over our external agent bridge, and formatted our trajectory exporter for NVIDIA NeMo Gym post-training. Our bridge protocol is framework-agnostic, so agents can also connect via plain Python or MCP.
  • NVIDIA Brev: We used L40S GPU instances on NVIDIA Brev for our early photo-to-3D reconstruction and texture pipelines before moving bursty live inference to Nebius Token Factory and on-demand GPU jobs to Nebius AI Cloud.
  • Tavily: Powers live web search when generating worlds and creatures (up to 3 searches per prompt), condensed into structured notes by Qwen3 235B on Token Factory.
  • 20 Hz Real-Time Runtime: Alongside the 1-minute turn loop, we built and benchmarked an in-memory 20 Hz (50 ms) spatial runtime in Python/FastAPI that steps 256 concurrent agents with a median tick under 2 ms in our benchmark test (well inside the 50 ms budget), pairing fast local steering reflexes with non-blocking background LLM calls.

Challenges & What We Learned

  1. JSON Schema vs. Forced Function Calling Across Models: We built a 22-case automated admission test before allowing any model to control a role in the world. That test revealed that Nemotron 3 Nano 30B used 61% fewer output tokens and answered more than twice as fast when prompted with a strict JSON schema (233 median tokens, 1.85 s median latency) compared to a forced function call (596.5 tokens, 3.95 s). Nemotron 3 Super 120B also required JSON schema mode to avoid generating spoken dialogue on silent movement actions.
  2. Budgeting for Always-On Reasoning Tokens: Because the Nemotron 3 and 3.5 endpoints always emit 150 to 800+ reasoning tokens before starting the JSON response, standard max_tokens limits (256 or 512) cut off the output before the answer began. We added per-role token floors (640 minimum up to 16,384) so reasoning never truncates the structured output.
  3. Fixing Multi-Turn Agent Loops in NeMo Agent Toolkit: Our external agent initially used the toolkit's ReAct agent, which formats prior tool calls back into the prompt as plain text. After one turn, the model copied that text format instead of issuing a fresh tool call. Switching to the toolkit's native structured tool_calling_agent resolved the issue and sustained 30-turn autonomous runs.

Current Limitations & What's Next

  • Procedural Parts Catalog vs. Open-Ended 3D Mesh Generation: Worlds and creatures are currently assembled by Nemotron from a procedural catalog of buildings, street grammars, and articulated 3D primitives; any requested details outside that catalog are listed on the creation card rather than generated from scratch. Next, we plan to pair our symbolic world state with generative 3D mesh pipelines and live neural shaders.
  • Stepped Simulation vs. Unified Real-Time Loop: The main town simulation runs in 1-minute steps (capped at 8 concurrent remote LLM-driven inhabitants per world to bound visitor token spend) and batch GPU jobs for painted views, while our 20 Hz 256-agent runtime and distillation exporter are ready to be unified into a single continuous real-time loop.

Try It Live

Built With

Share this project:

Updates

Submission history