Paradise

Stand still while the world changes: a 360° hillside generated live by Orbis that you reshape with a word, and Bosch's garden, where a pinch reveals what lies beneath. One WebXR page, one headset.

Comment

Inspiration

A generated world is only a place if most of it refuses to change. Real-time video models want to repaint everything, every frame; a place is exactly the part that must stay put while the sky, the weather and the light move.

That turned out to be the subject of both halves of our project. On the technical side we asked how to stand inside a world that a model is generating live, and keep the hill, the hut and the tree where they are while a single sentence turns the sky into something else. On the storytelling side we went to Bosch's Garden of Earthly Delights, where the same place can be read as paradise or as its consequence, and where, as the narration says, "the buildings remain; what changes is the way we see them."

What it does

Paradise is two 360° worlds in one WebXR page on Quest 3. You never move; the world does.

  • The Living Hillside — a panorama generated live by Orbis. Press a preset (Morning, Snow, Asteroids, Saturn, Rain) or type your own sentence, and the hillside becomes that scene: aurora over a ringed planet, asteroids streaking down with trails, snow under a full moon. Point at the sky and a ripple bends the picture where you point.
  • The Garden of Consequences — Bosch's garden as a painted world. Hold a pinch and a round opening follows your hand, showing the other reading of the same place. Day and night and the seasons pass through both worlds, with narration, score and in-world captions.

You switch between the two from a floating panel, without leaving VR.

The story we tell

Paradise is told in two movements.

The Garden of Consequences is the heart of it. Five hundred years ago Bosch painted a garden overflowing with pleasure. We place you in the middle of it and ask you to look closely, and eight short narration cues carry the arc:

  • Arrival — the garden as paradise, which may already contain the beginning of its undoing.
  • Reveal — hold a pinch, and the same place appears through another possibility. Is this a punishment, or a consequence we prefer not to see?
  • Return — let go, and the garden comes back. What you have seen remains part of it, even when it disappears.
  • Light — day passes into night in both worlds. The buildings remain; what changes is the way we see them.
  • Seasons — abundance has a season, and winter offers a pause. In the aftermath, rain crosses the ashes and snow settles over the damage: recovery as a question left open, not a promised ending.

The reveal is deliberately something you do, and can undo. The other world is never a cut or a level: it is under your hand for exactly as long as you hold it there, and the story is what you choose to keep looking at. Sound follows the same logic — the mix between the two worlds is driven by how far the reveal is open.

The narration draws on art historian Margaret Carroll's reading of Bosch through time and transformation, and it says plainly where we go beyond it: rest and recovery are our contemporary interpretation, not a claim about what Bosch intended.

The Living Hillside is the same idea in the present tense. It has no script. You say what the world should become — snow, a ringed planet, a sky full of asteroids — and it becomes that, while the hill, the hut and the tree stay where they were. One half asks what a place has been hiding; the other lets you find out what it could turn into.

In both, you stand still. The world changes around you, and the story is in what remains.

How we built it

A 360° world from a 16:9 video model. Orbis generates ordinary widescreen video and has no notion of a panorama. We take a 2:1 equirectangular panorama (World Labs Marble), squash it into 16:9 and give it to Orbis as its reference image; every frame it returns is therefore still a squashed panorama, and a fragment shader stretches it back over a sphere. The sphere is head-locked at infinity (view translation dropped, depth pinned to the far plane), the seam behind you is cross-blended, the poles fade to colours measured from the live frame, and the outer columns are cropped.

Re-anchoring instead of steering. A prompt alone cannot change the scene: the picture follows its reference image far more than its words. So every scene has its own reference, and a scene change is a new run on the same session: reset → set_image → set_prompt → start.

A two-session relay. Re-anchoring costs about nine seconds of black. We keep two Orbis sessions: one is on the sphere while the other is anchored and warmed up in the background, then the page cross-fades from one live video to the other and the roles swap. The same hand-over runs on a timer to return to the reference before the layout drifts, so nothing ever freezes. The fade only starts once frames of the new run have actually been decoded — the model's "frames emitted" message arrives before the frames do.

Custom prompts: paint, then direct. When you type a sentence, two models work in parallel. An image-editing model paints the request onto the scene's original reference in 3–4 seconds, keeping the fixed things in place. A small language model acts as a "prompt director": it returns structured JSON saying what moves, how, and what the light is like. Its answer is checked in code (no camera vocabulary, fixed things never listed as moving, a motion sentence must exist) before it becomes the Orbis prompt. Results are cached, and the new scene is handed to the relay.

The Garden: one decoder, two worlds. Garden and aftermath are two panoramas stacked in one 2048×2048 video, so a standalone headset decodes a single stream and the two worlds cannot drift apart. The reveal is a spherical cap measured as an angle around the pointing ray, so it is round wherever you look. Modes swap clips into the same video element; the sound layer mixes environmental beds by how far the reveal is open.

One page, one XR session. A page navigation ends an immersive session. A small shell owns the only renderer and XR session; each experience registers its own scene, camera and per-frame function, and a switch is a fade to black on a sphere that rides on the camera. An idle experience is really idle: leaving the hillside ends its live sessions, leaving the Garden pauses its decoder and sound.

Challenges we ran into

  • Only one of the two Orbis variants holds a reference. Documented as the same model, they behaved very differently: one matched our reference on the first frame (mean difference 5.7/255) and kept its camera still, the other re-composed the shot from the first frame under every prompt we tried.
  • A prompt is scene content, not an instruction. "The camera stays fixed on its tripod" put a tripod into the picture and the scene collapsed in ten seconds. Our prompts never name a camera.
  • Frozen asteroids. Our first custom-prompt pipeline described only weather and light, and its template said "everything stays exactly where it is" — so asteroids painted into the sky simply hung there. A motion sentence made them fly; leaving the word "time-lapse" in the prompt then buried the hill in smoke within twenty seconds. We ended up with a prompt in five parts: frame, layout, what stays, what moves, ambience.
  • Capacity. The model we depend on can serve very few sessions at once, so the page has to cope with being refused: it runs one line and retries, falls back to another model, and recovers sessions that end underneath it.
  • Two apps, one headset session. Both experiences listen to the same XR controllers and draw on the same canvas; every input handler has to know whether its world is the one on screen.

Accomplishments that we're proud of

  • A live, generated 360° world that you can stand in on a standalone headset, and reshape with a sentence in about twelve seconds.
  • Scene changes and the periodic return to the reference with zero frozen frames.
  • Two very different experiences — a technical study and a piece of storytelling — fused into one page and one VR session, with a 1–2 second switch.
  • Every number in our write-up was measured: we built a headless test harness that records frames and scores layout and seam drift.

What we learned

  • Anchor a video model with pictures, not words; use words for what moves.
  • Say what must stay as explicitly as what must change.
  • Do not trust a model's own report of what it has done — neither "frames emitted" nor a language model's JSON. Verify both.
  • In VR, what the viewer sees during a wait matters more than how long the wait is.

What's next for Paradise

  • Voice input in the headset, so custom scenes do not need a keyboard.
  • Checking the seam behind the viewer on painted scenes, and rejecting pictures that move the horizon.
  • Letting the Garden's worlds be regenerated live under the painted architecture, with the same relay.

Credits

Real-time video: Orbis by Visko on Reactor. Panorama: Marble by World Labs. Custom scenes: Gemini image editing and Claude Haiku. Narration voice: ElevenLabs. Score: Suno. The Garden draws on Hieronymus Bosch's The Garden of Earthly Delights and cites Margaret Carroll's reading of Bosch through time and transformation; where the experience extends that reading, it says so.

Built With

Share this project:

Updates

Submission history