SIMULACRA — VR Teleport

VR Teleport fixes VR calls by teleporting your whole body right into someone's home. Summon someone into your environment or teleport into theirs, fully, using live 3D.

Inspiration

Human connection is essential. For the past half century, people have relied on phone calls to stay close to loved ones who live far away, whether it's a child who went off to college or a friend who moved across the country for a job. But phone calls have a problem: besides the laggy sound, the awkward interruptions, and the echoes of your own voice, you cannot see the other person.

Then came Zoom, FaceTime, and Skype, and we could finally see our loved ones through a small screen. VR brought us a little closer, but depending on the platform, calling someone in VR usually means talking to a floating head or a Wii-like avatar instead of the real person. No matter how good the camera or connection is, the person on the other side still feels separated from you rather than actually present in the same space.

So we asked a different question: why can't we just teleport the person? SIMULACRA is our answer. It is a spatial telepresence platform that combines real-time video calling, stereoscopic camera capture, and virtual reality to create the feeling of being transported into another person's environment. Instead of simply seeing someone through a screen, our goal is to make remote communication feel like actually being there with them.

What it does

VR Teleport is an immersive video calling platform that allows two users to communicate through a normal desktop browser or a Meta Quest 3 headset. You put on the Quest, open our web app, and choose one of two options:

Summon invites someone into your environment.
Teleport requests to virtually enter their environment.

Once a Summon is accepted, you see a live stereoscopic 3D representation of the other person right in front of you. Ask them to reach toward you, and their hand actually comes toward you with real depth, because each of your eyes is receiving a different perspective of them.

The platform supports traditional flat video calls alongside 360° and 180° 3D viewing modes. For our stereoscopic setup, we used two NexiGo N60 webcams positioned like human eyes to capture slightly different perspectives of the same scene. These two feeds are combined into a side-by-side video stream, with one half representing the left eye and the other half representing the right eye. When the user enters VR, VR Teleport separates the two perspectives and displays the correct image to each eye, creating stereoscopic depth and making the call feel more natural and immersive than a traditional video call.

The person on the other end doesn't have to be family. It could be a realtor walking you through a house, a tailor fitting you for a suit, or a business partner overseas you need to negotiate with. Have them in your space, or go into theirs, fully.

How we built it

We created our repository and divided the project into a frontend and backend. We built the frontend using React, Vite, Three.js, React Three Fiber, WebXR, and WebRTC. We first implemented the basic video calling system, including login, contacts, incoming and outgoing calls, camera selection, microphone controls, and camera controls. We then added WebRTC to transmit live audio and video between users with low latency. Our frontend supports Flat, 360°, and 180° 3D viewing modes, and the remote WebRTC stream is passed directly into our VR scene for rendering.

For the immersive portion, we used Three.js and React Three Fiber to convert the incoming video stream into a video texture and render it inside a virtual environment. The two NexiGo N60 webcams capture separate left and right perspectives, which we combine into a side-by-side stream. We then modified our renderer so each eye samples only its own half of the frame. For a horizontal texture coordinate $u \in [0, 1]$ on the rendered surface, the sampled coordinate $u'$ is:

$$ u' = \begin{cases} \dfrac{u}{2}, & \text{left eye} \[8pt] \dfrac{u}{2} + \dfrac{1}{2}, & \text{right eye} \end{cases} $$

Our original 180° renderer only displayed the left half of the side-by-side stream, so implementing true stereoscopic rendering became an important part of our development.

We built the backend using Node.js, Express, Socket.IO, and JWT authentication. Socket.IO handles user presence, Summon and Teleport invitations, WebRTC offers and answers, and ICE candidates, while the actual audio and video are transmitted through WebRTC rather than being processed by our backend. We also added STUN and TURN support so our application can establish connections even when a direct peer-to-peer connection is blocked by the network.

Challenges we ran into

Our first major challenge was getting true stereoscopic video working inside the Meta Quest. Our original 180° mode could display a side-by-side video, but both eyes were effectively seeing the same perspective, which meant the video looked immersive without actually creating stereoscopic depth. We had to change the way our Three.js renderer handled the incoming video so that each eye received a separate perspective.

Our next major challenge was working with the two NexiGo N60 webcams. The cameras had to be positioned and aligned closely enough for the left and right images to create a comfortable 3D effect. Perceived depth depends on the disparity between the two images, following the standard stereo relationship

$$ Z = \frac{f \cdot B}{d} $$

where Z is the depth of a point, f is the focal length, B is the baseline (distance between the two cameras), and $d$ is the horizontal disparity between the left and right images. Because of this, small differences in camera angle, position, and field of view introduce unwanted disparity, misaligning the two images and reducing the quality and comfort of the experience.

Another major challenge was networking. WebRTC works best when two devices can establish a direct peer-to-peer connection, but some Wi-Fi networks prevent that. We added TURN support so calls could fall back to a relay server when a direct connection was not possible. We also had to make sure our application was served over HTTPS, because browser camera access and WebXR require a secure connection on devices such as the Meta Quest. Our Vite configuration uses HTTPS and proxies API and Socket.IO traffic to the backend.

Accomplishments that we're proud of

We are proud that we were able to combine several different technologies into one working prototype. We successfully built a real-time video calling platform that supports authentication, online contacts, Summon and Teleport calls, WebRTC audio and video, Meta Quest WebXR support, multiple viewing modes, and stereoscopic video rendering.

What we are most proud of is taking a normal video call and turning it into something that can actually be experienced inside VR. Seeing a teammate reach toward the cameras and watching their hand come toward you with real depth was one of the most important milestones for us, because it showed that our idea of a more immersive form of remote communication was possible using consumer hardware and web technologies.

What we learned

We learned that building immersive telepresence requires several different technologies to work together. Throughout the project, we gained experience with WebRTC, Socket.IO, WebXR, Three.js, React Three Fiber, stereoscopic vision, camera alignment, STUN and TURN networking, and Meta Quest development.

We also learned how important communication and teamwork are when building a project that combines hardware and software. Each part of VR Teleport depended on another part of the system working properly, so every member of our team had an important role in testing, debugging, and integrating the project. Most importantly, we learned that creating a sense of presence does not require expensive specialized equipment. By combining consumer webcams, real-time communication, stereoscopic vision, and virtual reality, we built a prototype that moves beyond the experience of a traditional video call.

What's next for SIMULACRA

VR Teleport is built on a strong foundational prototype with significant potential for expansion into a complete spatial telepresence platform. Our next focus is making it possible for two people to walk around the same room together, evolving beyond stereoscopic video into true six-degree-of-freedom telepresence. We want to use multiple camera perspectives to estimate depth and reconstruct the remote environment using technologies such as depth maps, point clouds, 3D meshes, or Gaussian splatting. This would let users move around the reconstructed environment rather than only rotating their head inside a stereoscopic video.

We also plan to expand with real-time person reconstruction, body tracking, facial reconstruction, spatial audio, and room scanning. That could mean a parent spending time with a child hundreds of miles away, friends separated across the country sharing a room again, or doctors, designers, realtors, and businesses interacting remotely as though they were physically together. Our long-term vision is to reconstruct both the person and the place so a remote user feels they have actually entered another person's environment.

FaceTime brought someone's face to you. We want to bring the person to you.