Orbis Engine

real video vs 4d render
multiple rendered clips, with object detection (strong daniel benchpress)
meta quest, og video, and live view observability all at once
admin panel trace observability (graph version) - analyze pipeline & catch bottlenecks
admin panel trace observability (list version) - analyze pipeline & catch bottlenecks

Inspiration

We've always wanted to watch movies like we were really there, part of the experience, in first person. The Orbis Engine lets us do just that, and more.

What it does

Orbis turns an ordinary video into a 3D world you can explore. Put on a VR headset and you're in the room, in first person, while the moment plays out around you exactly as it was recorded. You can talk to people, pick up objects, and switch scenes without taking the headset off. It works on your favourite movie scene, an old video on your phone, a big moment from last night's Raptors game, or a workout tutorial where you can walk around the trainer and check their form.

How we built it

V0, first steps: rebuild the room from the video. We started with Depth Anything 3, VGGT and Brush. It looked great from the original camera position but not anywhere else, because nothing filled in the space the camera never saw. People were worse: V-DPM and PIFuHD gave us melted limbs.

V1: keep the rebuilt room, generate the missing views. We tried Stable Virtual Camera, GeoNVS, GEN3C and Lyra. The outputs included a courtyard that didn't exist, grey mush, a real shop sign rewritten into gibberish, and a scene too slow to walk through.

V2: generate the whole room, then correct it with the footage. This is the pipeline we shipped:

Camera: Pi3X works out where the camera was in every frame.
Room: we mask the people out, fill the holes with LaMa, and Marble generates a full Gaussian-splat room from the cleaned video. Then we train it against the real frames so the made-up parts get replaced by what was actually recorded.
People: each person is tracked and rebuilt as an LHM avatar, driven by their motion in the video. We tried correcting the poses against the footage and it made them worse, since one camera can't tell you how far away a hand is. So we left them alone.
Placement: we fit scale and floor so everyone's feet land where they actually stood.

V3: put you inside it. Three.js with Spark for the splats, WebXR for Quest, and the source video as the master clock so audio and motion stay in sync. You walk at eye height instead of flying a camera, a spectator view shows everyone else what the headset sees, and in one scene you can talk to a character and pick up what they're holding.

How we used OpenAI, Codex and Devin

Devin built the front end. It made the landing page section by section (hero, gallery, compare, how it works) and restyled the demo shell three times until we liked it, each as its own PR we could review and merge while the rest of us stayed on the pipeline.

OpenAI API is in the product in four places:

Describing the scene: GPT looks at frames from the video and writes the prompt we send to Marble.
Checking the output: GPT looks at renders of each generated room and throws out the broken ones.
Getting the scale right: GPT finds objects with a known size, like a door, so we know how big the room really is.
Talking to people: in VR you can walk up to someone and have a voice conversation with them through the Realtime API.

Codex ran the long jobs for us. We gave it a goal and a budget, and it worked for hours on our cloud machines: launching GPU runs, checking results, fixing failures, trying the next idea:

Research. Measured why rebuilding the room fails (a forward-moving camera only sees about 41° to each side), tested view-generation models that didn't work, and ran the fine-tuning that made generated rooms match the real footage.
Evaluation. Tested whether Marble does better with full video than still frames. Ran the paid pipeline on two clips, compared against the source, and rejected both: one room had fixtures that weren't in the video, the other a warped bench. Kept the older versions and documented why.
Safety. Built a ledger of every paid GPU call so crashed runs resume instead of billing twice, plus a gate that checks every generated room against the real frames before it reaches the demo.
Integration. Merged everyone's branches into main and wrote tests for what it moved.

There are a hundred years of film and a few trillion phone videos in the world. We want to open all of them up in 4D.