[NOT] AURA - AI-Driven Urban Safety

• 

• 

Inspiration

Critical information is often available during an emergency or tactical operation, but it is fragmented across drone feeds, robots, cameras, maps and mobile reports. Operators must constantly switch between screens while losing awareness of where detected people or hazards are located relative to their own position.

We wanted to turn this distributed sensor data into an intuitive, heads-up XR experience. Instead of only showing another camera feed, [NOT] AURA places detected entities directly in the operator’s spatial view—even when a wall or another obstacle blocks the operator’s line of sight.

The project is inspired by tactical-awareness platforms, drone reconnaissance and the idea of giving first responders, infrastructure inspectors and remote operators an “X-ray-like” understanding of their surroundings.

What it does

[NOT] AURA – Tactical XR Awareness is a distributed perception and augmented-reality system that allows an operator to visualize people and other points of interest through walls.

External sensors—such as an OAK depth camera mounted on a robot, drone or remote observation point—detect people and calculate their 3D positions. These positions are transmitted through ROS 2 and rendered as spatial markers in XREAL glasses. As a result, the operator can see the approximate location of a remotely detected person even when that person is hidden behind an opaque obstacle.

The current prototype can:

• detect people with YOLO,
• estimate their distance using OAK depth data,
• convert detections into 3D ROS 2 markers,
• align the virtual scene with the operator’s head orientation,
• display detected entities as red spatial markers,
• stream remote camera and desktop views,
• accept markers from other ROS 2 nodes through a common interface,
• control a Unitree G1 using head and hand movement,
• operate in simulation before connecting to physical hardware.

The broader architecture is designed to connect mobile incident reports, databases, tactical-awareness platforms, drones, Nav2-enabled robots and remote cameras with an edge computer and an XR headset.

How we built it

We built the prototype as a modular ROS 2 system using Python and C++.

An OAK camera provides synchronized RGB and depth images. A YOLO model detects people in the RGB stream, while the corresponding depth region is used to estimate their distance. The system converts every detection from image coordinates into a position in the XR scene and publishes it as a ROS 2 visualization marker.

XREAL IMU data is received over TCP and processed with a Madgwick filter. This provides the operator’s head orientation, allowing the virtual camera and spatial markers to respond naturally when the operator looks around.

The teleop_moving_window package creates the XR-style scene, renders video streams and receives markers through /xreal/virtual_scene/add_block. This interface allows detections to come not only from the local OAK camera, but eventually from drones, robots and other distributed sensors.

The system also includes:

• ROS 2 launch orchestration,
• low-latency camera and desktop streaming,
• hand and head tracking for Unitree G1 control,
• MQTT and WebRTC communication experiments,
• SROS 2 enclaves and mutual-TLS certificates,
• Raspberry Pi and external-computer integration.

Our current hardware prototype uses XREAL glasses connected to a separate compute unit running the ROS 2 and AI stack.

Challenges we ran into

The hardest challenge was spatial alignment. Detecting a person in a camera image is relatively straightforward, but placing that person correctly in the operator’s XR coordinate system requires combining camera geometry, depth measurements and headset orientation.

Depth data can be noisy, especially at longer ranges or around the edges of a person. We implemented filtering and movement limits to reduce sudden jumps in the displayed markers.

Latency was another major challenge. Camera capture, AI inference, depth processing, ROS 2 communication and XR rendering all add delay. Even small delays become noticeable when the operator moves their head.

We also had to work with different coordinate systems, resolutions, update rates and communication protocols. Keeping the camera, IMU, robot and virtual scene synchronized required careful debugging.

Finally, the current prototype depends on a relatively heavy, tethered external computer. This proves the concept but limits mobility—the exact problem we want to solve in the next hardware iteration.

Accomplishments that we're proud of

We created a working proof of concept that transforms a person detected by an external RGB-D camera into a spatial marker visible in an operator-controlled virtual scene.

We are especially proud that the system is not limited to displaying a conventional video feed. It extracts the detected person’s approximate 3D position and represents that information independently, creating the foundation for through-wall tactical awareness.

We also built a common ROS 2 marker interface, meaning future drones, robots, applications or tactical platforms can publish information into the same XR scene without rewriting the visualization system.

Additional accomplishments include:

• real-time YOLO person detection with depth estimation,
• head-controlled XR visualization,
• low-latency visual streaming,
• integration with a physical Unitree G1,
• natural hand-to-arm and head-to-torso control,
• simulation and hardware launch modes,
• independently enabled safety-critical controls,
• experiments with secured distributed communication.

What we learned

We learned that “seeing through walls” is primarily a distributed sensing and spatial-registration problem. The glasses do not need to physically sense through a wall if another trusted sensor can observe the hidden area and share the detected object’s position.

We also learned that visual stability is as important as detection accuracy. A marker that jumps, drifts or responds incorrectly to head movement can be more distracting than useful.

Building the prototype showed us how important common coordinate frames, timestamps and clearly defined ROS 2 interfaces are when combining XR, computer vision, robotics and remote sensors.

We also discovered that the best tactical interface should not overwhelm the operator. Future versions must communicate confidence, distance and source information without covering the real-world view with unnecessary graphics.

What's next for [NOT] AURA - Tactical XR Awareness

Our next goal is to transform the current proof of concept into a fully mobile, multi-sensor tactical-awareness platform.

We plan to:

• replace the tethered compute box with on-device processing using XREAL Aura and the Snapdragon XR platform,
• integrate detections from drones, mobile robots and fixed cameras,
• connect the system with TAK-style situational-awareness infrastructure,
• use Nav2 maps and shared world coordinates for more accurate marker placement,
• add persistent tracking when a target becomes temporarily occluded,
• replace prototype cubes with clear silhouettes, icons and confidence indicators,
• visualize hazards, navigation paths and reported incidents in addition to people,
• improve latency, calibration and multi-camera sensor fusion,
• evaluate the system with first responders, inspectors and remote operators,
• add privacy controls, access permissions and secure data sharing.

The long-term vision is a lightweight XR system that allows an operator to understand what is happening beyond their direct line of sight by combining information from every available sensor into one spatial view.

Built With

• ai
• atak
• c++
• kalman
• nav2
• ros-2
• slam
• web-app
• xreal