[LA 32] Seekr: Navigational Intelligence inside World Models

XGRID scan 01
XGRID Scan 02
character with visual perception 01
character driven by the chat
objects identified by the character
collision mesh
the World model generated by a prompt
correct object identification
character created with Tripo AI
character

Inspiration

Most games ask the player to control a character directly. We asked a different question: what if the character were an autonomous AI agent, and the player had to coach it?

ScavengeAI grew from our interest in spatial intelligence—the ability of AI systems to perceive, understand, remember, and act within three-dimensional environments. We wanted to explore this through a playful experience built around human-AI collaboration rather than direct control.

What it does

ScavengeAI is a scavenger hunt inside a Gaussian Splat reconstruction of a real-world space.

The game is played by autonomous Explorer avatars powered by a navigational brain. An Explorer can perceive objects and landmarks, reason about their locations, build a mental map, and move through the environment toward a goal.

The human player acts as a coach. Instead of using a keyboard or joystick, the player communicates through natural-language chat. The player can assign a goal, suggest a direction, or help the Explorer recover when it becomes confused.

Together, the human coach and AI Explorer discover landmarks, search for hidden objects, and complete challenges.

How we built it

We reconstructed the environment as a Gaussian Splat and rendered it as an interactive 3D world using Spark and Three.js. A corresponding collider mesh provides the geometry needed for movement and collision detection.

Our navigational architecture separates the Explorer’s capabilities into several layers:

Visual perception identifies objects and landmarks in the Explorer’s field of view. Spatial grounding connects perceived objects to locations in the 3D environment. Spatial memory records visited places, landmarks, and navigable relationships. An AI action planner interprets the player’s request and selects the Explorer’s next action. The movement system translates those decisions into navigation through the scene.

This architecture allows us to change or improve individual perception, planning, and navigation models without rebuilding the entire experience.

Challenges we ran into

Gaussian Splats provide visually detailed reconstructions, but they do not automatically provide the clean geometry, semantic labels, or navigational structure an autonomous agent needs.

One major challenge was connecting what the Explorer sees in a 2D camera view to a useful position in 3D space. Recognizing a bottle is only the first step; the agent must also determine where the bottle is, whether it can reach it, and how to travel toward it without colliding with the environment.

We also had to balance the reasoning abilities of vision-language models against the latency required for an interactive game. Autonomous navigation needs a combination of fast perception, higher-level reasoning, spatial memory, and reliable low-level movement.

Accomplishments that we're proud of

We created an interactive AI Explorer inside a Gaussian Splat environment and developed a modular navigational brain that connects perception, spatial understanding, planning, and movement.

We are particularly proud that the human is not simply issuing movement commands. The player and Explorer have different roles: the AI perceives, remembers, plans, and moves, while the human provides goals, strategy, and corrections through conversation.

This creates a different form of gameplay based on coaching an autonomous character rather than controlling one.

What we learned

We learned that object recognition alone is not enough for spatial intelligence. An autonomous agent must connect semantic understanding—“this is the object I need”—to geometry—“this is where it is and this is how I can reach it.”

We also learned that spatial memory does not need to begin as a complete metric map. A lightweight topological representation of places, landmarks, and connections can give an agent useful memory while it explores.

Most importantly, we found that the imperfections of an AI agent can become part of the game. The player’s role becomes meaningful when the Explorer can act independently but still benefits from human guidance.

What's next for ScavengeAI

Next, we plan to develop ScavengeAI into a multiplayer experience where multiple human coaches and AI Explorers can play together. Explorers could share discoveries, divide scavenger-hunt tasks, compete or collaborate, and build a collective understanding of the environment. We plan to give Explorers longer-term spatial memory, more reliable obstacle avoidance, and the ability to learn from previous exploration sessions. We also want to introduce multiple Explorers that can share discoveries, divide search tasks, and collaborate with human coaches.

Beyond gaming, the same technology could support architecture and digital-twin applications. Autonomous agents could explore proposed or existing spaces, test wayfinding systems, simulate how people navigate buildings, and help designers evaluate airports, museums, campuses, and other complex environments before changes are built.