Inspiration
AlphaStar and OpenClaw: I'm interested in two things in this study: the right scaffolding for an LLM to Model the World, and finding emergent behaviour through networked agents not initiated by a system prompt. I'm looking to build an "inverse WestWorld", where the agents are given no script, and instead find purpose through the right tool calls (the model) to understand and take action in the world.
What it does
CrabRaver understands the world around it, knows who is in the same server room as it, can interact with other agents in a 3D environment. The agents can speak, chat, and express their emotions on their avatar. They can also shoot each other with laser guns.
How we built it
The agent is fed a multilayered data in tiers around its environment in a token efficient manner. This works well and is agnostic to polygons OR splats, such as those from WorldLabs. Obstacle information is fed through a voxel grid, and the agent can pick a point on the map, which then uses pathfinding instead of a traditional dual joystick navigation.
Challenges we ran into
Specific to this hackathon, there is an effect of lossiness when the camera is in proximity to a splat, so while it is technically correct, it is visually poor, akin to enlarging a raster image. Even in a 2M splat, the tools do not exist that had been rolled out with polygon interaction over the past decades. It is very nascent as of yet, and I'm looking forward to tools such as camera culling, LODs, occlusion, and culling. I was also not able to use the Pico emulator, as it was silently failing when sideloading APKs. I would like an equivalent of adb or similar to tell me why it's failing to sideload. The latter is far from a showstopper, it is merely something that I chose not to overcome to focus on content.
Accomplishments that we're proud of
DigitRaver was able to display splats despite its customized rendering stack, I was not sure if I'd be able to accommodate. CrabRaver was also able to recognize the splats through screenshots and make meaningful remarks around its environment.
What we learned
Gaussian Splats, Pico.
What's next for CrabRaver
CrabRaver is an idea I'm going to continue to pursue. There's multiple angles of exploration, including prompt tweaking, better world model understanding and perhaps just the idea of time. I'm also looking forward to the evolution of splats as well as a DigitRaver VR port.
Log in or sign up for Devpost to join the conversation.