Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for AgentDeck XR
The problem. AI agents got powerful, but they still live in a chat tab: a wall of text you scroll past, with no sense of who is working, what they decided, or why. When an agent runs multi-step work for you, the process is a black box. Observability shouldn't live in a log file — it should live in your space.
The solution. AgentDeck XR is a spatial agent workbench for Meta VR. You step into your workspace and find a three-agent crew standing next to you: Planner, Runner, and Fixer. Each has a glowing 3D avatar that shows its live state — bright while working, dim at rest. You send an intent ('Plan my day', 'Add a task', or anything you type), and the crew delegates: Planner reasons over the request and drafts a plan, Runner executes it as spatial tool calls, and Fixer verifies the outcome and closes the loop. Task cards materialize on a 3D rail in front of you — amber when queued, pulsing blue while acting, green when done. Every step streams into a trace on the spatial console, so you can stand next to your agents and literally watch them think.
Crew architecture. The agent core (src/agent/) is zero-dependency TypeScript: an event-driven intent → plan → act → trace state machine with a bounded queue, a tool registry whose tools return typed scene commands (plan_day, spawn_task, clear_tasks, report_status), and a pluggable LLM provider speaking OpenAI-compatible /chat/completions in JSON mode. Every failure path — missing key, timeout, HTTP error, bad JSON — degrades to a deterministic mock planner, so the experience never dead-ends. A thin AgentConsoleSystem bridges runtime events to the 3D world: avatars, cards, and console text are pure projections of the same event stream, which also feeds the trace ring buffer.
How we used IWSDK. AgentDeck XR is built on IWSDK (Immersive Web SDK) with its Three.js Entity Component System, the Horizon spatial UI kit for the console panels, UIKitML for panel markup, and the built-in IWER XR Simulator (Meta Quest 3 profile) for desktop testing. Development runs through the IWSDK CLI (npm run dev, HTTPS dev session, emulator auto-injection), and npm run build ships a static site to GitHub Pages — the judge link is the production build, no local setup required. Components like CrewCard and systems like AgentConsoleSystem follow IWSDK's ECS patterns end to end.
Why it fits Agentic Interaction. The interaction model is delegation, not prompting: humans state goals; visible agents negotiate the plan, act on the world, and report back. State is legible at a glance (avatars, card tints, crew status rows), inspectable in depth (role-prefixed trace lines), and extensible (new tools are new scene commands — the same loop could drive voice input, which is already wired to dispatch through the identical path).
What's next: hand-claim verification of spawned cards on device, voice capture on Quest Browser, and multi-crew scenes.
Live demo: https://skye-cqu.github.io/agentdeck-xr/ — Source: https://github.com/skye-cqu/agentdeck-xr
Built With
- github
- iwsdk
- meta-quest
- three.js
- typescript
- uikitml
- vite
- webxr
