Your Immersive & Interactive Portfolio Website

Inspiration

JB Proxie, my locally-hosted AI agent trained on my career and research, already answers questions on my portfolio site — but only through a chat window. This hackathon was the chance to give him a body: turn a text-based agent into a spatial one, and turn my resume into a place instead of a page. My career spans both XR and AI, so building the first version around my own story was the natural proof of concept.

What it does

It turns a career into an explorable, conversational portfolio. Visitors walk through five Gaussian-splat worlds spanning my life — a Royal Netherlands Air Force hangar, a research lab recreating my Master's and PhD work (including a playable mini-game controlled by real gaze-tracking and hand-flex, the same mechanics from my dissertation), a holographic AfterNow stage, a mountaintop VR studio reached by putting on a virtual HTC Vive, and a datacenter where four projectors sum into a portrait of me — a playable version of the optical-computing research I did at Microsoft. JB Proxie walks alongside you throughout: aware of what you're looking at, answering in character, and able to teleport you between chapters just by talking to him. It runs on desktop, mobile, and VR headsets, and it's built explicitly as a template — proof that any professional's resume could become this.

How we built it

The frontend runs on IWSDK + SparkJS 2.0 Gaussian splats, Vite, and TypeScript, with full WebXR support for Quest. World Labs Marble generates the environments, Tripo3D generates individual hero props, and Mint handles pre-visualization, audio, and UI content. JB Proxie runs Qwen3 35B locally on my NVIDIA DGX Spark, with a FastAPI backend and RAG over my published research and resume. I worked solo, pairing with two persistent AI agents inside Claude Code — one focused on world design and generation prompts, the other on XR performance — and tracked every asset and decision in a shared spreadsheet to keep the build coordinated overnight.

Challenges we ran into

Coordinating three generative platforms, each suited to different assets, without knowing the exact token cost of any of them up front. Keeping the embodied 3D companion's behavior — short spoken replies, scene-teleport commands — cleanly separated from the existing 2D chat widget on my site, since both run on the same backend. Staying strictly accurate about NDA-protected work while still making it feel real and interactive. And budgeting performance carefully enough that a live render-to-texture mini-game doesn't blow the frame rate on Quest.

Accomplishments that we're proud of

Getting the datacenter reveal to actually work: four projectors, individually just noise, summing in real time into a recognizable photo — a playable analog of real optical-computing research, not a metaphor for it. A companion who feels present without being intrusive, with real personal-space logic. A faithful recreation of my actual PhD research mechanics — gaze-based camera steering and hand-flex locomotion — as something you can play, not just read about.

What we learned

That the hardest part of a multi-platform generative pipeline is coordination, not generation — a shared decision tracker mattered as much as any prompt. That real reference material makes AI-generated worlds far more convincing than invented abstraction. And that giving an AI collaborator a narrow, well-defined role produces sharper results than one generalist trying to do everything.

What's next for Immersive & Interactive Portfolio Website

Finishing generation now that every design decision is resolved and all three token pools are loaded. A real headset pass to verify voice, hand tracking, and gestures on-device. Upgrading JB Proxie from a 2D billboard companion to a fully rigged, animated avatar. And the actual long-term goal: turning this from my portfolio into a real template anyone could use to make their own.

Built With

• chatgpt
• claude
• dgxspark
• marble
• mint
• quest
• tripo3d
• vscode
• webxr
• worldlabs