Hero

Step inside a world that Visko Orbis generates live, and steer it from VR with your Meta Quest 3 controllers.

Comment

What it does

Hero streams a world that Visko Orbis generates live into a WebXR scene in Meta Quest Browser. You stand in VR facing a 16:9 window onto the generated world. Aim a Quest controller at a control, pull the trigger, and Orbis receives a new prompt on the same live session (no restart). The world morphs at the next chunk boundary.

How we built it

  • Next.js + Reactor's official visko-orbis-stable reference app (server-side session JWT broker; the API key never reaches the browser).
  • The main_video WebRTC track from the typed Orbis provider is bound to an app-owned <video> and rendered as a Three.js VideoTexture on a plane in an immersive-vr session.
  • Quest Touch Plus ray + trigger (select) sends set_prompt; the cooldown and duplicate guard stop overlapping sends.
  • A built-in diagnostics layer (desktop panel + in-VR HUD) tracks render fps, decoded fps, texture uploads/s, and WebRTC stats, so we can tell a rendering freeze from a slow stream.

Verified on a real Quest 3

  • Live Orbis video on the VR plane at the full 18 fps stream rate; XR render ~79–90 fps on the 90 Hz display.
  • Three controller-trigger prompt changes on one session, all acknowledged by the model; no restart.
  • 94 s in XR with 0 s texture stall.

Challenges

  • The headset's USB stayed in Quest Link mode (no ADB), so we served the app over an HTTPS tunnel instead. WebXR only needs HTTPS.
  • Stream delivery sometimes drops from 18 fps to a few fps after ~30–60 s. Our metrics show this is network/stream delivery, not rendering. 1080p delivery helps.

What's next

Turn the two test controls into a story-driven sequence of world changes, and stabilise stream delivery for the demo.

Built With

Share this project:

Updates