Inspiration
What if a neural network could become a computer?
Before you open an app on your phone, you can already picture it: where the buttons are, what happens when you tap one, what the next screen will be. Nothing is compiled in your head. You learned it by watching and using it.
What it does
On the left is a real computer running deterministic code. On the right is the same computer imagined by a world model: no OS, no apps, window manager Each frame, the model sees only the current screen, your mouse and keyboard input, and a small memory it writes for itself. From that it predicts the next frame, then feeds that prediction back in, again and again. Everything you see on the right, including Doom, is dreamed.
How we built it
Every screen is a sequence of tokens. We render the screen as a 32×24 grid of 8×8-pixel tiles, and each tile is a token from a fixed catalogue: a pair of letters, an icon corner, a piece of a card. A whole frame is 768 tokens, so predicting the next screen becomes predicting 768 tokens.
Doom works differently. Its view is split into 160 thin vertical columns, and each column is described by a single token: which wall it is, how tall it looks, and its texture. A few extra tokens mark where enemies are. The model predicts these tokens, and a simple renderer turns them into the picture you see.
To teach it, we needed a lot of examples, so we wrote a scripted user that uses the computer the way a person would. It moves the mouse in curved, slightly shaky paths, clicks with human timing, and does actions like opening a file, typing a note or hunting enemies.
The model is a 12-layer transformer that predicts the entire next frame at once. Alongside the screen it keeps a small set of memory tokens that carry over from frame to frame, so it can remember things you can't see, like where you're standing in Doom. It also answers questions about the computer's hidden state, such as which app is open or how much health you have, which forces it to keep track of what's really going on.
