Inspiration
Low Earth orbit is getting crowded fast: there are now more than 10,000 active satellites and tens of thousands of tracked debris objects. Starlink alone reports tens of thousands of collision-avoidance maneuvers every six months, and that number roughly doubles each period. Today most conjunction screening runs on threshold rules (maneuver when \( P_c > 10^{-4} \)) with human operators in the loop. That approach doesn't scale, and every unnecessary burn spends \( \Delta v \), which shortens a satellite's life. If avoidance fails, the debris can set off a Kessler cascade that closes whole orbital shells. We wanted to find out whether an RL agent could learn better maneuver policies than hand-tuned rules.
What it does
Kessler is a reinforcement-learning environment and mission-operations dashboard for autonomous collision avoidance in LEO. An agent controls a satellite through a series of debris conjunctions. It has to keep the miss distance outside a keep-out zone, while uncertain about where the debris really is, and spend as little \( \Delta v \) as possible while staying in its assigned orbital slot. The dashboard runs everything in the browser:
- a WebGL globe with about 6,000 debris objects propagated on the GPU
- a live B-plane view and a Hill-frame trajectory plot
- a benchmark against baseline policies that reruns on every page load
- live policy training across 8 parallel Web Workers
How we built it
Environment (core.js): It uses the same reset() / step() API as Gymnasium. Relative motion follows the Clohessy–Wiltshire equations:
$$ \ddot{x} = 3n^2x + 2n\dot{y} + a_x, \qquad \ddot{y} = -2n\dot{x} + a_y, \qquad \ddot{z} = -n^2 z + a_z, \qquad n = \sqrt{\mu / r^3} $$
They are propagated with the closed-form state-transition matrix \( \Phi(\tau) \). Debris arrives at relative velocities of 2–14 km/s, and each encounter is scored in the B-plane, the standard frame for real conjunction analysis.
Uncertainty model: The tracking uncertainty \( \sigma \) shrinks from 400 m to 25 m as time to closest approach decreases. Collision probability \( P_c \) comes from the Rice distribution, evaluated with Bessel-function quadrature.
Observation and reward: The agent sees a 28-dimensional observation vector. It includes a physics-informed feature, the STM sensitivity
$$ \frac{\partial \lvert \text{miss} \rvert}{\partial \Delta v} = \Phi_{rv}(\tau)^{\top} \hat{m} $$
which tells the agent how much a burn would change the miss distance. The reward penalizes keep-out violations, closeness to the debris, \( \Delta v \) spent and drift from the orbital slot.
Policy: A 2,083-parameter MLP (\( 28 \to 32 \to 32 \to 3 \), tanh) trained with OpenAI-style Evolution Strategies: antithetic sampling, rank-based fitness shaping and the Adam optimizer. Training is distributed the way Salimans et al. describe, so workers exchange only random seeds and scalar fitness values.
Baselines: (1) no maneuvers; (2) an operator-style rule that triggers at \( P_c > 10^{-4} \) at a 25-minute decision point, then flies a CW two-impulse transfer back to the slot.
Frontend: Debris orbits are propagated in a WebGL vertex shader. Benchmark episodes and the ES workers run in Web Workers, and the build step bakes the trained weights.json into a single index.html.
Challenges we ran into
- Credit assignment over long horizons: A burn made 25 minutes before closest approach only pays off much later, and it changes both along-track and radial drift. A plain observation vector wasn't enough for the policy to learn this reliably, and adding the STM sensitivity feature is what made learning work.
- The early-vs-late trade-off: Burning early is cheap but based on an uncertain estimate. Burning late is certain but costs much more \( \Delta v \). Shaping the reward so the agent learned that balance, without collapsing to "never burn" or "always burn," took many iterations.
- Doing it all in the browser: Propagating about 6,000 objects in real time while also running 900 benchmark episodes and 8 training workers meant moving the orbit propagation into shaders and keeping heavy computation off the main thread.
- Fair evaluation: We held out 300 scenarios (about 1,040 conjunctions) for testing, so results reflect how well the policy generalizes, not memorized training scenarios.
Accomplishments that we're proud of
- A physically grounded environment: CW/STM dynamics, B-plane geometry and Rice-distribution \( P_c \), not a toy 2D simulator.
- On held-out scenarios, the RL agent has the best overall return:
| Policy | Violations | Mean \( \Delta v \) | Slot error | Return |
|---|---|---|---|---|
| No maneuvers | 43.8% | 0 m/s | 482 m | −70.0 |
| Operator rule | 1.7% | 2.96 m/s | 501 m | −33.6 |
| RL agent (ES) | 5.9% | 3.82 m/s | 296 m | −20.0 |
It cut violations 7× compared with not maneuvering, learning only from reward.
- Distributed ES training runs live in the browser: the training curve passes the no-maneuver baseline in about a minute.
What we learned
- Orbital mechanics: relative motion in the Hill frame, why an along-track burn produces radial and along-track drift (the \( 3\,\Delta v\,\tau \) rule), and B-plane conjunction geometry.
- Conjunction risk: computing \( P_c \) from Gaussian position uncertainty, and why operators use a \( 10^{-4} \) threshold.
- RL engineering: physics-informed features in the observation can decide whether a policy learns at all, and reward shaping matters as much as the learning algorithm. For cheap simulators, ES parallelizes much better than policy-gradient methods.
- Systems: GPU particle propagation in GLSL shaders, and using seeds to communicate between parallel workers.
What's next for Kessler
- Close the violation gap: the hand-tuned operator rule still has fewer keep-out violations (1.7% versus 5.9%). Next we'll try residual RL on top of the rule-based controller, train with PPO, and run longer training.
- Higher-fidelity dynamics: \( J_2 \) perturbations, atmospheric drag and real TLE/CDM data from Space-Track.
- Multi-agent scenarios: coordinated maneuvers when both objects are active satellites, such as two constellations meeting.
- Packaging: a Python port that registers with Gymnasium so other RL researchers can benchmark on it.