Rover XR (RXR)

Rover XR Title Slide
Rover XR Details
Architecture And Tech Stack
The team!
Galaxy-XR ADB Building
Unity Development
Headset in action
Rover Flashing and Low Level Scripting
The BEAST
BOO! JUMPSCARE!!!
can you believe this guy?

What Started It

This project is the culmination of a process that began with a simple conversation about how useful it would be to control a robot via a VR headset. What started as a “that would be cool” idea quickly turned into something we genuinely wanted to build.

So we decided to give it a shot!!

We used the tech at the Infer-X Lab, and a lot of caffeine and adrenaline, to build this crazy ambitious project.

What it does

Our project enables users to control a physical Wave Rover through an XR interface, leveraging edge technology for real-time interaction. Commands are processed on the Rubik Pi and sent directly to the rover through a UART interface for quick, low-latency wireless navigation.

On top of that, the Rubik Pi runs YOLOv8 object detection on the live camera feed and streams annotated video back to the headset, giving the operator real-time visual awareness while driving.

How we built it

Our hardware stack utilized the Galaxy-XR headset, a Rubik Pi, and the Waveshare Wave Rover.

On the software side, we used Unity to create the virtual interface. Using C# scripts in Unity, we send UDP packets of interface interaction to the Rubik Pi, which hosts its own network for quick edge processing. On the Rubik Pi running Ubuntu Server, we parse the interaction packets and send JSON movement requests to the rover MCU, which operates the vehicle.

For vision, we integrated YOLOv8 directly on the Rubik Pi to perform real-time object detection on the camera feed. The Pi annotates the video with bounding boxes and streams it back to the Galaxy-XR headset via MJPEG.

Challenges we ran into

Networking and hardware hell!!! 💀

Flashing Rubik Pis took a lot of work to get going. This was our first time working with Rubik Pis, and we had to read a ton of documentation and do a lot of testing to make sure the components we needed were compatible and would work the same way they do on a Raspberry Pi, which we're more familiar with.

We also had trouble managing real-time video streams, integrating multiple hardware components, and handling UDP packets alongside IP addresses, port numbers, and DNS masking services.

Our biggest challenge was engineering the full loop between the XR headset and the rover-Pi combo. We had to switch protocols multiple times for our video stream to ensure full compatibility with our XR interface, especially once we added real-time object detection at the edge.