Terrain

• 

  us trying to setup isaac sim at 3 am

• 

  collecting free merch ;)

• 

  the llamas helping us keep our cool

Inspiration

2 months ago, Amogh's (hacker in Terrain) family's farmhouse in the Nilgiris was caught in a fiery forest fire and got burnt to ashes. They weren't able to salvage anything from the house was because it was on a steep slope of the mountains, making it harder for firefighters to carry necessary relief goods and packages. Mechanisms to efficiently and successfully transport heavy relief and industrial goods that were designed to be carried by humans across a host of unpredictable and dangerous terrains portray poor real-world performance. We wanted to change this, and hence Terrain was born. Terrain is a proprietary world model built to help facilitate successful time-sensitive transport and delivery of critical goods to terrains and environments that are dangerous/inaccessible to humans.

What it does

The world model assesses the current state of the robot with the sensitive goods it's supposed to carry and the goal location where the goods are to be transported. Based on the trained world model, it dictates the robot's actions and motions to ensure that the robot is able to carry necessary essential goods across unpredictable terrains to a goal location. The robot dynamically adjusts the orientation of the load it's holding when traversing through uncertain territory to ensure that the overall stability of the system is maintained.

How we built it

We built a custom Predictive World Model in two stages. We trained an MLP-Policy based on Reinforcement Learning on existing locomotion policies for the Unitree G1 to improve its locomotive performance in unpredictable and uneven terrains. We reduced fall rate on uneven terrain by 70.59% and a 600% increase in duration of transport until failure.

After this, we took various states of the Unitree G1 with a heavy load on uncertain terrain and trained a CNN-based World Model. The robot dynamically adjusts the orientation of the load it's holding when traversing through uncertain territory to ensure that the overall stability of the system is maintained.

We noticed over a 7x improvement in the median distance traversed with the load, marking a significant improvement over standard robot policy.

Challenges we ran into

It was quite hard to setup the Nebius environment to access Isaac Lab and Isaac Sim for obtaining training data. The Nebius VM was quite hard to setup, but we were able to manage it at the end. Obtaining relevant data from Isaac Lab and Isaac Sim was quite annoying too, and validating our model required making custom eval tests. We tried different types of model architectures, everything from vJEPA to transformer-based models. Setting up our environments and resolving all dependencies to effectively train the models was time-taking.

Accomplishments that we're proud of

We're proud of making significant progress and developing a novel existing robotic policy, given that no one in our team had previous experience working in World Models for Robotics. Given that we not only successfully trained an RL and World Model but also made significant improvements on existing proprietary robot policies.

What we learned

We learned a lot about predictive and generative world models and building the entire neural network end-to-end, all the way from synthetic data generation using a Physics Engine (Isaac Sim and MuJoCo), setting up the base architecture for the model and receiving relevant outputs. We also learnt about robotic policies and training such policies such that they are deployment-ready on actual humanoid robots.

What's next for Terrain

We would expand on the possible use cases for Terrain and train it on more edge use cases accounting for various types of loads and various types of terrains.

Built With

• isaac-lab
• isaac-sim
• mlp
• mujoco
• nebius-physicalai-workbench
• ppo
• rl
• world-model