Hazard Intelligence

Inspiration

When one of our team members was 8 in the Poconos, he went off trail, and was face down in the snow for 20 minutes. He was rescued by a helicopter, and then someone skied down from a safe point to where he was. Any longer, and there could have been fatal consequences.

But he’s not the only one. French ski slopes see 40,000+ rescues every winter, US national parks logged 2,890 search and rescue missions in a single year, and an average US winter kills 27 people in avalanches. Rescue work is dangerous and constant.

Humanoid robots have made some substantial mechatronics progress and learned zero-shot policies on a lot of simple tasks, like cooking, but none of that shows how a person moves on ice, on variable terrain where the real dangerous tasks lie. Humanoid robots could one day go where responders can't, but they learn to move from human motion captured in studios, on flat floors. We used known terrain connected with human data to build datasets and train policies for skiing and hiking robots. Millions of people already hike and ski with a phone in their pocket, and the apps they use record where they went but not how their bodies moved. As Olympic recordings and human ski training webcam games. already out there.

What it does

We trained a policy that teaches a humanoid to ski. Using reinforcement learning (PPO) on 4,096 simulated robots at once, a Unitree G1 learned to ski real World Cup pistes in physics, with ski physics we built ourselves. On a benchmark of 24 slopes (8 to 31 degrees, real course sections plus synthetic ones), the pretrained controller alone finished 1 of 24 without a fall (4%). With our policy, it finishes 19 of 24 (79%) in under 24 hours. It runs through a Model API anyone can call.
Our game collects data from frequent skiers. Using MediaPipe, you can ski a real piste in front of a webcam, and the humanoid learns from you. Skiers can use it to preview a slope before embarking in real life, starting with the Streif at Kitzbühel, and 47 courses from 26 resorts being available. Using Presage’s SmartSpectra, our webcam reads heart rate and breathing to identify, on average, the more physically demanding parts of the path.
Hikers and skiers contribute from their phones. Our biggest leverage point is data from people already skiing and hiking the exact trails we want to model. By partnering through trail apps, ski resorts, and passes, individuals can opt in easily and reliably with Photon’s iMessage agent. After we receive consent, whenever a user enters a geofence for a trail head, our iOS app records accelerometer and gyroscope at 100 readings a second, GPS, barometer and altitude. This data informs much of the individual run data to train our model.
Together, a data layer for rescue robots. Data from all three forms lands in one database tagged with the exact trail, slope and effort, allowing a rescue humanoid to train before going. Even if we capture 0.1% of the hiker/skier user base that records a 2-hour bike ride, we’d get ~120,000 hours of human movement on real terrain, over 500 times the 220 hours in the standard motion-capture archive robots train on today.

How we built it

The phone side is a native iOS app (SwiftUI, Core Motion, Core Location with background mode) that uploads each hike in the background. A Python server matches it to OpenStreetMap trails and elevation data and extracts per-second features like pace, climb, slope and cadence.

A Photon Spectrum iMessage agent runs the whole conversation: opt-in, consent, each person's own link, and opt-out flow.

Everything lands in one SpacetimeDB database: the live game data, the courses and terrain, every individual user recording, and the policy registry.

For the robots, NVIDIA GEAR-SONIC provides the body controller and our policy sits on top, trained in MuJoCo Warp on an H100 through Modal. Ski courses are built from OpenStreetMap and elevation tiles, and the Model API runs on Modal.

Challenges we ran into

Our first phone recorder was a web link, but iOS freezes web pages the moment the screen locks, and a hike means hours with the phone in a pocket. We rebuilt it as a native app that stays alive through background location.

We had to run additional tests to verify that our sensors were picking up motion correctly. Checked noise and drift with the phone lying still Compared step count tracker against 100 manually counted steps Verified gyroscope recognized three full turns (1,080 degrees) Climbed lengths of known elevation to check GPS distance and barometer

Accomplishments that we're proud of

Our ski policy went from 1/24 clean runs to 19/24 on the benchmark slopes in >24 hours
The policy is zero-shot, hardware-agnostic, meaning it can easily work across any other humanoid with a similar form factor.
We tested our app on a real trail this morning to find accelerometer, gyroscope, and other metrics were accurate
The whole loop works on a real phone: opt in by text, hike with the screen locked, upload, match to the trail, store, and text the hiker a summary.
Everything lives in one database: 26 resorts, 47 ski courses, hiking courses with 1 m lidar, every recording and our policy registry.
Our Model API's outputs match the training code to within 4e-6.

What we learned

The hard part of training rescue robots is retrieving data. Real terrain data from real people only scales if you leverage a pre-existing user base rather than trying to grow one from scratch. We also learned that consent and verification have to be built in from the start, including transparency about what data is being recorded from a user’s phone and the process for deleting it.

What's next for hazard intelligence

The next step is securing partnerships that allow us to mass-collect user trail and ski data. For trails, we plan to approach apps like Strava, AllTrails, Gaia GPS, Komoot and Garmin Connect. For skiing: resort apps and pass programs like Epic Mix (Vail), Ikon Pass (Alterra) and Slopes. Partners would offer incentives in return for opting into our training data initiative.

Even if we capture 0.1% of the userbase of the hiking app AllTrails and record a 2-hour bike ride, that’s 120,000 hours of data collected on a real trail, which is x500 better than the 220 hours motion-capture archive robots train on today.

Rescue teams, national park services, and ski patrol would also be able to access this data with live conditions and a hazard map.

We also hope to tackle other physical tasks like rock-climbing, swimming, and mine navigation.

Together, we hope to build the data infrastructure that trains the future of humanoid rescue.