Shadowplay

Inspiration

Games have always been a proving ground for AI, and 3D games especially: they’re where agents have to learn space, not just pixels. The frontier is impressive. OpenAI’s VPT learned Minecraft from tens of thousands of hours of video, DeepMind’s DreamerV3 mined diamonds by learning a world model from scratch, and NVIDIA’s Voyager built a lifelong-learning agent on top of game APIs. But every one of these needs massive compute, enormous datasets, complex training pipelines, or privileged access to the game’s internals. None of it is something you can just plug into a game and use.

We’re interested in the opposite direction: lightweight agents that rival frontier systems through clever engineering rather than scale. The hard version of that dream is an agent that plays a 3D game just by watching your screen. That’s what our earlier project, GhostPlayer, attempted, and it taught us exactly where it breaks. A single monocular view makes depth unreliable, every decision waits on a slow model round-trip, and the model has no body: no sense of where it is, what it’s holding, or whether its last action worked.

So for Shadowplay we started from the other end. Before teaching a lightweight agent to see the world, we wanted to build the body it should eventually drive. We chose Minecraft as our first sandbox because it’s the most widely shared playground in world-model research. It’s a fully 3D, procedurally generated, physically consistent world, and everyone, researchers and players alike, already understands it.

What it does

Shadowplay is an AI agent that plays Minecraft as you, controlling your own character rather than a spawned bot. You type a goal in chat, like “get me a stone pickaxe” or “go into that cave, light it up, and mine some coal.” The agent breaks it into a plan, narrates the plan in chat, and executes it at real player speed: chopping trees, crafting through the actual inventory screens, digging for stone, diving into a lake to fetch sand, walking into a cave and placing torches as it gets dark.

The moment you touch your keyboard or mouse, it stops and hands control back. One command later, it picks up where it left off, re-planning from wherever you are now.

How we built it

Shadowplay is a client-side Fabric mod for Minecraft 1.21.1 with three layers:

Brain. Gemini turns a plain-English task into a structured plan over a fixed catalog of skills, validated before anything runs. Plans can also be pinned for deterministic replay. Body. Baritone handles pathfinding and mining. Our own code handles aiming, block placement, and crafting through real inventory slot clicks, so it all happens visibly, the way a player would do it. Senses. Instead of guessing from pixels, the agent reads exact ground truth: its position, the block it’s targeting, light levels, and inventory. Every skill declares a postcondition the executor checks before moving on, so it never assumes an action worked.

Perception is deliberately geometric. Finding a cave means treating underground air as a space and extracting its connected components with a flood fill. Torch placement is a reflex driven by measured light, not a model call.

Challenges we ran into

The hardest problem was handing control between the human and the agent. Baritone overrides player input while it drives, so naively you end up fighting your own character. We built an interrupt system that reads physical key and mouse state directly, distinguishes your input from the agent’s own clicks in crafting menus and from you typing in chat, and stops cleanly instead of yanking you back.

Crafting was the second wall. There’s no “craft” API on the client, so the agent operates the real crafting UI slot by slot, with pacing so it stays watchable and verification that the right item came out.

The third challenge was determinism. A demo that works four times out of five isn’t a demo. Every skill got timeouts, retries, and checkable success conditions until the same task produced the same result every run.

Accomplishments that we’re proud of

Watching the agent go from an empty inventory to a stone pickaxe from a single sentence, with the whole tech tree planned and executed, never stops being satisfying. Even more so the moment it walks into a dark cave and starts lighting its own way.

We’re also proud of the architecture’s philosophy: the model decides what to do, and a harness that understands the world decides how. That split is what turned GhostPlayer’s one-block-every-three-seconds into real-time play. And the whole thing runs locally on a laptop, costing pennies per task.

What we learned

The bottleneck for game-playing agents often isn’t intelligence; it’s embodiment. Given a reliable body and honest feedback about the world, a general-purpose model plans surprisingly well. Without them, even a strong model flails. Strict structure helps: constrained skills, validated plans, and postconditions turn an unreliable model into a reliable system.

We also learned what good ground truth is worth. Once you can read exactly where everything is, you can see precisely what a vision-only agent would get wrong, and that’s the starting point for fixing it.

What’s next for Shadowplay

The harness is step one. Because Shadowplay can read exact depth, position, and targets for every frame the player sees, it can act as a teacher: generating perfectly labeled data to train and evaluate lightweight monocular perception models, so the same skills can eventually run from screen pixels alone. From there, we want to explore topological methods for spatial reasoning, like connectivity of free space and navigability, as cheap ways for small models to verify their surroundings without solving full depth estimation.

Afterwards, we're interested in designing agents that can drop into any 3D game, learn its body, and play.