WINJAY-XR

WINJAY XR turns multi-agent AI into a living, spatial world,step inside a mind map that grows around your question, where evidence becomes objects, conflict becomes physical tension.

Comment

Inspiration

Every multi-agent AI demo looks the same: a wall of chat where models talk to each other and the human is left scrolling. We asked a different question: "why would a human ever read what agents say to each other?"

So we flipped it. "Models talk to models. Humans watch the state." Agent communication becomes a "spatial state machine" you can stand inside, where meaning has a physical form: evidence is an object, a hypothesis is a node, conflict is physical tension, and a contradiction is a collision.

Meta Quest and mixed reality were the natural home for this. Reasoning has shape, and a headset is the first interface that can show it.

What it does

WINJAY XR has two worlds built on one shared truth layer.

"State Machine world." A Researcher, Analyzer, Falsifier, and Decision agent work on a hypothesis. Evidence appears as objects, and support is a green curve. A contradiction is a jagged red edge that makes the hypothesis tremble. The final decision lands as a green ACCEPT or red REJECT box.

"Living Mind Map (hero demo)." A mind map grows around one question, ""3-Day Vacation on a Limited Budget"", built by three agents with visible personalities:

  • "Explorer" (blue) grows branches: Transport, Lodging, Activities, Food, Budget.
  • "Connector" (green) finds distant links, drawn as teal bridge arcs and ghost suggestions.
  • "Challenger" (orange) marks tensions as red vibrating edges.

You aren't a user, you're a "curator". You can "prune" weak nodes, "lock" a decision gold so agents must work around it, or "speak" an idea into the map, where your words become a node weighted higher than any agent's. Every action triggers real agent reactions. On Quest, you do all of this with hand-tracking pinches, and a scrubber lets you rewind the whole timeline.

How we built it

"WINJAY CORE is the truth layer." It's a state machine, event bus, transition validator, provenance tracker, and delta-state journal, written in C# (netstandard2.1) with no Unity dependency. Web, desktop, and Unity are just three renderers of the same state.

Agents never mutate state directly:

Agent -> PROPOSE -> State Engine -> VALIDATE -> COMMIT

Stale proposals, illegal transitions, and unknown entities are rejected, and the same event order always produces the same state.

"Strength is arithmetic, not animation." After every commit it's recomputed from the log:

$$ \Delta s \;=\; 0.15\,N_{\text{support}} \;-\; 0.20\,N_{\text{conflict}} \;+\; 0.05\,N_{\text{bridge}} $$

Human-created links count $\times 1.5$, and locked nodes are frozen. Rewinding time is just folding a shorter prefix of the journal.

"The stack:"

  • "Core + tests:" C# with 70 automated xUnit tests.
  • "Mock agents:" deterministic scripts behind an AgentProposal interface that stays the same when real LLMs replace them.
  • "Web gateway:" ASP.NET Core Kestrel + WebSocket with a Three.js viewer, and an isolated world per connection.
  • "XR:" Unity 6 LTS + Meta XR SDK 71 + Meta XR Simulator, with passthrough MR and hand tracking. The engine runs inside Unity from the exact same sources.

Challenges we ran into

  • "The spec didn't match its own demo." Our scenario needed extra supporting evidence and a contradiction arriving after a hypothesis was already SUPPORTED. We added two transitions, SUPPORTED -> SUPPORTED and SUPPORTED -> CONFLICTED.
  • "Alive but honest." Everything visible had to come from the event log, so nothing could be a decorative animation.
  • "XR without a headset." We built an offline compile check that builds all 48 Unity scripts with Roslyn against the real Unity 6 reference assemblies, with package APIs mirrored as stubs and cross-checked against the Meta XR SDK 71 source.
  • "Scope discipline." We cut voice input. The mock agents are deterministic scripts, so a voice command had nothing meaningful to drive, and the Meta Voice SDK can't be tested without a headset.
  • "A presenter-proof demo." The story auto-runs, but a presenter can perform the curator steps live, and the scripted fallbacks detect the world state and skip themselves.

Accomplishments that we're proud of

  • A deterministic, validated truth layer shared by web, desktop, and XR.
  • A mind map where strength, ripples, and rewind are real computation, not animation.
  • A human role built on participation (prune, lock, speak) instead of prompting.
  • 70 passing tests and an offline compile check covering the full Unity layer.

What we learned

  • Making agent reasoning visible as "space" changes how you trust it. A contradiction you can see collide is easier to understand than one buried in a log.
  • Separating "propose" from "commit" makes multi-agent systems debuggable and safe, and lets mock agents be swapped for LLMs without touching anything else.
  • An event log gives you rewind, replay, and provenance almost for free.
  • Cutting a feature you can't honestly demonstrate beats faking it.

What's next for WINJAY-XR

  • Replace the mock agents with real LLMs behind the same AgentProposal interface.
  • Add voice interaction for speaking ideas into the map.
  • Test and polish on real Quest hardware.
  • Add more scenarios beyond the vacation map and the state machine.

Built With

  • .net
  • agentic-ai
  • ai-agents
  • c#
  • deterministic-simulation
  • hand-tracking
  • human-ai-interaction
  • meta-quest
  • meta-xr-sdk
  • meta-xr-simulator
  • mind-map
  • mixed-reality
  • multi-agent-ai
  • openxr
  • productivity
  • spatial-computing
  • three.js
  • unity
  • websockets
  • xr
Share this project:

Updates

Submission history