Inspiration

Low Earth orbit is getting crowded fast: there are now more than 10,000 active satellites and tens of thousands of tracked debris objects. Starlink alone reports tens of thousands of collision-avoidance maneuvers every six months, and that number roughly doubles each period. Today most conjunction screening runs on threshold rules (maneuver when \( P_c > 10^{-4} \)) with human operators in the loop. That approach doesn't scale, and every unnecessary burn spends \( \Delta v \), which shortens a satellite's life. If avoidance fails, the debris can set off a Kessler cascade that closes whole orbital shells. We wanted to find out whether an RL agent could learn better maneuver policies than hand-tuned rules.

What it does

Kessler is a reinforcement-learning environment and mission-operations dashboard for autonomous collision avoidance in LEO. An agent controls a satellite through a series of debris conjunctions. It has to keep the miss distance outside a keep-out zone, while uncertain about where the debris really is, and spend as little \( \Delta v \) as possible while staying in its assigned orbital slot. The dashboard runs everything in the browser:

  • a WebGL globe with about 6,000 debris objects propagated on the GPU
  • a live B-plane view and a Hill-frame trajectory plot
  • a benchmark against baseline policies that reruns on every page load
  • live policy training across 8 parallel Web Workers

How we built it

Environment (core.js): It uses the same reset() / step() API as Gymnasium. Relative motion follows the Clohessy–Wiltshire equations:

$$ \ddot{x} = 3n^2x + 2n\dot{y} + a_x, \qquad \ddot{y} = -2n\dot{x} + a_y, \qquad \ddot{z} = -n^2 z + a_z, \qquad n = \sqrt{\mu / r^3} $$

They are propagated with the closed-form state-transition matrix \( \Phi(\tau) \). Debris arrives at relative velocities of 2–14 km/s, and each encounter is scored in the B-plane, the standard frame for real conjunction analysis.

Uncertainty model: The tracking uncertainty \( \sigma \) shrinks from 400 m to 25 m as time to closest approach decreases. Collision probability \( P_c \) comes from the Rice distribution, evaluated with Bessel-function quadrature.

Observation and reward: The agent sees a 28-dimensional observation vector. It includes a physics-informed feature, the STM sensitivity

$$ \frac{\partial \lvert \text{miss} \rvert}{\partial \Delta v} = \Phi_{rv}(\tau)^{\top} \hat{m} $$

which tells the agent how much a burn would change the miss distance. The reward penalizes keep-out violations, closeness to the debris, \( \Delta v \) spent and drift from the orbital slot.

Policy: A 2,083-parameter MLP (\( 28 \to 32 \to 32 \to 3 \), tanh) trained with OpenAI-style Evolution Strategies: antithetic sampling, rank-based fitness shaping and the Adam optimizer. Training is distributed the way Salimans et al. describe, so workers exchange only random seeds and scalar fitness values.

Baselines: (1) no maneuvers; (2) an operator-style rule that triggers at \( P_c > 10^{-4} \) at a 25-minute decision point, then flies a CW two-impulse transfer back to the slot.

Frontend: Debris orbits are propagated in a WebGL vertex shader. Benchmark episodes and the ES workers run in Web Workers, and the build step bakes the trained weights.json into a single index.html.

Challenges we ran into

  • Credit assignment over long horizons: A burn made 25 minutes before closest approach only pays off much later, and it changes both along-track and radial drift. A plain observation vector wasn't enough for the policy to learn this reliably, and adding the STM sensitivity feature is what made learning work.
  • The early-vs-late trade-off: Burning early is cheap but based on an uncertain estimate. Burning late is certain but costs much more \( \Delta v \). Shaping the reward so the agent learned that balance, without collapsing to "never burn" or "always burn," took many iterations.
  • Doing it all in the browser: Propagating about 6,000 objects in real time while also running 900 benchmark episodes and 8 training workers meant moving the orbit propagation into shaders and keeping heavy computation off the main thread.
  • Fair evaluation: We held out 300 scenarios (about 1,040 conjunctions) for testing, so results reflect how well the policy generalizes, not memorized training scenarios.

Accomplishments that we're proud of

  • A physically grounded environment: CW/STM dynamics, B-plane geometry and Rice-distribution \( P_c \), not a toy 2D simulator.
  • On held-out scenarios, the RL agent has the best overall return:
Policy Violations Mean \( \Delta v \) Slot error Return
No maneuvers 43.8% 0 m/s 482 m −70.0
Operator rule 1.7% 2.96 m/s 501 m −33.6
RL agent (ES) 5.9% 3.82 m/s 296 m −20.0

It cut violations 7× compared with not maneuvering, learning only from reward.

  • Distributed ES training runs live in the browser: the training curve passes the no-maneuver baseline in about a minute.

What we learned

  • Orbital mechanics: relative motion in the Hill frame, why an along-track burn produces radial and along-track drift (the \( 3\,\Delta v\,\tau \) rule), and B-plane conjunction geometry.
  • Conjunction risk: computing \( P_c \) from Gaussian position uncertainty, and why operators use a \( 10^{-4} \) threshold.
  • RL engineering: physics-informed features in the observation can decide whether a policy learns at all, and reward shaping matters as much as the learning algorithm. For cheap simulators, ES parallelizes much better than policy-gradient methods.
  • Systems: GPU particle propagation in GLSL shaders, and using seeds to communicate between parallel workers.

What's next for Kessler

  • Close the violation gap: the hand-tuned operator rule still has fewer keep-out violations (1.7% versus 5.9%). Next we'll try residual RL on top of the rule-based controller, train with PPO, and run longer training.
  • Higher-fidelity dynamics: \( J_2 \) perturbations, atmospheric drag and real TLE/CDM data from Space-Track.
  • Multi-agent scenarios: coordinated maneuvers when both objects are active satellites, such as two constellations meeting.
  • Packaging: a Python port that registers with Gymnasium so other RL researchers can benchmark on it.
Share this project:

Updates

Submission history