The view from the website.
The view inside VR.
Viewing and interacting with the live stream from the browser.
Architecture

Inspiration

Over the last few years, remote and virtual collaboration has exploded as we open up industries to collaboration from around the world and become ever more concious the cost commuting and long-distance travel has on our environment. We were inspired by the need to bridge the gap between virtual expertise and physical application. We envisioned a tool that brings expert guidance directly to the field, no matter the distance.

For instance, a technician repairing airplane could receive real-time instructions and supervision from a remote expert, with AR markers guiding the process. Or, surgeons or medical students could benefit from real-time, AR-enhanced guidance during intricate and specialised procedures or training sessions even in remote communities.

What it does

InstructAR is an innovative VR app that streams live camera feeds from a VR headset to a website, allowing remote users to place interactive AR markers in 3D space. These markers appear in real-time in a VR user's field of view, guiding them through complex tasks with precision and ease.

VR View

How we built it

We built InstructAR using a combination of technologies and frameworks. The VR encoding and streaming uplink is powered by Python with OpenCV and NumPy, while the AR overlay is implemented using Unity (C#) and native Meta Quest 3 APIs. The front-end of the website was developed with TypeScript using Next.js (React), and the back-end integration was achieved with Node.js running TypeScript behind Cloudflare and an NGINX reverse proxy. Each of these 4 applications communicates with each via Socket.IO except for the headset and the steaming uplink which communicate via ADB.

Architecture

Challenges we ran into

Throughout the development of InstructAR, we encountered several challenges:

Real-Time Streaming Latency: Ensuring low-latency, high-quality video streaming was critical. We tackled this by establishing low-latency connections via Socket.IO from the browser to the server and the server to the headset video processing uplink. Using compression and variable frame timing, we are able to stream frames within milliseconds to a remote browser anywhere in the world.
AR Marker Accuracy: Achieving precise alignment of AR markers in the VR environment required an intricate understanding of realtime 3D to 2D projection and vice-versa. This involves raycasting into a 3D mesh of the real world, using the headset's position, orientation, marker location on the 2D plane and corrections for rotation and lens distortion of the video livestream.
User Interface Design: Designing an intuitive UI for both web and VR users presented a unique challenge as InstructAR needs to make collaboration easier, not harder. This means designing a system no hidden tricks and a simple interface.

Accomplishments that we're proud of

Successfully integrated real-time AR markers into a live VR stream.
Developed a user-friendly interface that allows seamless interaction between web and VR users.
Achieved low-latency streaming, enhancing the real-time collaboration experience.

What we learned

This project was a tremendous learning opportunity for us. We gained insights into:

The complexities of raycasting and mapping 2D position to 3D locations in VR.
The importance of UX design in mixed-reality applications.
Advanced techniques in latency sensitive video streaming and multicasting.

What's next for InstructAR

Looking ahead, we plan to:

Expand the functionality of InstructAR to support customisable markers, text annotations and a native voice link.
Enhance the scalability of our platform to accommodate larger user bases.
Explore partnerships with headset and wearables creators such as Meta, Apple and Rayban to broaden the application scope and platform availability of InstructAR.

We are excited about the potential of InstructAR to transform remote collaboration and bring expert knowledge to where it's needed most.

Built With

Submitted to

UNIHACK 2024

Created by

I wrote the web frontend, backend (with Jordan), created the branding, UI designs and created the pitch. I had a lot of fun streaming and displaying realtime video on the web!

Angus Bosmans
I wrote most of the backend (TS) and capture (Python) code, and helped kick off the ideation phase. Unfortunately, Meta doesn't expose any camera data through their APIs, so a lot of my time was spent finding ways around this ;))

Jordan Zdimirovic
I wrote the AR application. Was a bit of a struggle getting the AR marker to be placed where the webapp user clicks on their 2D screen both accurately and consistantly but got there in the end.

Jason Fooks