Search&Rescue AR

In disaster zones and rural s&r missions, responders juggle radio, maps, thermal cameras, and medical tools while visibility is low and environment noise is high.

Search&Rescue AR is a zero-latency AR system. It overlays thermal-based target detection (with bounding boxes), patient vitals, and GPS/Wi-Fi positioning directly in a passthrough HUD so rescuers can see people, health data and location without taking their hands off the mission.

We combined an AR passthrough HUD built into the XREAL Air Pro 2s with a thermal imaging pipeline that runs detections and draws bounding boxes around heat signatures. These boxes are streamed to a custom app that allows for true passthrough using the XREAL glasses, enabling true zero-latency environment viewing. We also integrated vitals from a PPG sensor into the HUD widget system.

Challenges we ran into

• We did not have a bi-directional USB-C to DP cable which forced us to stream IR data from the Jetson to our laptop, introducing latency.
• YOLO-V8 person tracking model did not work as well as we wanted for the use case, so had to pivot to alternate detection methods.

Accomplishments that we're proud of

• Working target detection and full integrated HUD with live updating GPS location as well as patient health data.

What we learned

• We learned a lot about server-client networking and vision based detection models. We also learned about passthrough and true AR.

What's next for Search&Rescue AR

• Spend more time on hardware packaging.
• Get correct cables to reduce latency.
• Integrate wireless connection with health sensors.

Built With

• heartrate
• heatmap
• hud
• ir-camera
• jetson-nano
• numpy
• open-cv
• python
• ultralytics
• xreal
• yolo