HyperSoldier

A MR concept for headset-enhanced combat forces. Proof that today's off-the-shelf commercial hardware and software can already be used to prototype high impact, multi-input human enhancing services.

Comment

Teaser Video | GitHub Repository

Inspiration

Inspired by Anduril's EagleEye (video showcase), Palantir's collaboration with Meta on IC2 (video showcase, presentation and more), and Rivet's Hard Spec.

Quentin Valembois' recreation of Meta's new AI glasses on Quest (original post here) was what made me believe something similar to his work could be pulled off to prove that today's off-the-shelf commercial hardware and software can already be used to prototype high impact, multi-input human enhancing services.

What it does

When equipped with the required gear, users can track their location, get assisted threat detection, be aware of their surroundings and more. All hardware components can be integrated into standard military gear (helmet, uniform, etc.). All software runs offline, with devices communicating with each other over their own local network.

How to try it

After installing the app on Meta Quest, you need to also install the companion app, HyperCompanion, on an Android mobile device. The latest release is available here.

Once installed, provide location and camera access, enable your GPS and start broadcasting. Make sure both your Android device and your Meta Quest are connected to the same network. HyperSoldier assumes that the Android device will be placed behind your head, pointing back, this ensures proper compass and rear camera view behavior. It is recommended that you first start HyperSoldier on your Quest and only then start HyperCompanion on your mobile device.

To toggle the rear camera view in the HUD, nod your head vertically. To toggle the map as well as threat detection, nod your head horizontally. Note that world maps for tile mapping do not exist for free at a good enough resolution and the resources that are publicly accessible end up taking significant space (in the order of tens of GB and more). So, for this release, only Portugal (my home country) has map coverage - all other locations will show up blank in the HUD's map.

How I built it

The main application is built with Unity, targeting Meta Quest 3/3S. The companion application is built with Kotlin, targeting Android. The wearable band application is built with Kotlin, targeting WearOS (under development, not included in this release).

These apps involve the use of the following technologies: Unity Inference, Meta SDK, Meta Camera Access API, OpenXR, WebSocket, WebRTC, UDP, and more.

Challenges I ran into

There are multiple challenges when aiming at achieving the full potential of what is here proposed.

The main challenges faced so far were:

  1. Optimizing speed and accuracy of threat detection in real-time, given that it's fully running locally in headset.
  2. Achieving adequate visual location display, given that open-source world maps have too low resolution and online services were not an option.
  3. Designing UI and interactions for a context where controllers are unavailable, hands are busy with other tasks, voice might put the user at risk and eye gazing is not hardware-supported.

Accomplishments that I am proud of

The obvious first achievement is the mere fact that it was actually possible to implement this initial baseline work that, in our eyes, proves the original thesis and excites us to improve the prototype and explore the concept further.

On a more practical / technical level, the main accomplishments to be proud of are:

  • Effective low-latency real-time video streaming for in-headset soldier rear view
  • Meaningful use of Meta's Passthrough Camera API and AI-powered real-time object detection
  • Clever HUD control interactions through head nods (vertical and horizontal)

What I learned

It always takes longer to implement than originally planned.

Beyond all of the new tech (zero prior knowledge of Kotlin, Meta's PCA, and Unity's Inference / Sentis), I did learn that Quest hardware does still impose significant processing limitations. I couldn't get above 60fps with object detection running locally, the field of view is too narrow to be worth wearing it in a combat situation and passthrough creates significant image distortions in fast paced situations and outdoor environments.

What's next for HyperSoldier

Oh there's so much more that can be done for HyperSoldier... Here's just a few from the long list:

  • non-simulated health tracking
  • ammo tracking (through shot audio detection or other reliable approach)
  • silent gesture-to-command translation
  • scene understanding for danger evaluation / path risk heatmap
  • scene understanding for cover / concealment object identification
  • drone control (with aerial view, similar to rear view)
  • integrated radio communications
  • digital laser designator and danger area pinpoint sharing
  • battlefield alerts (checkpoint arrival, enemy down, etc.)
  • squad members location tracking
  • squad members 6dof (through-wall) tracking

Built With

Share this project:

Updates