Astrophage

Two-Stage Random Forest Classifier Model for NASA Kepler Object of Interest (KOI) Exoplanet Validation

Comment

Inspiration

The Kepler telescope found thousands of planet candidates, but most of them aren't actually planets. They're false positives: binary stars, instrumental noise, random stellar activity. And every time astronomers chase a false positive, they waste telescope time that could've gone to a real planet.

We started wondering: how do actual astronomers decide what's real? Turns out, they don't just look at a signal and say "planet or not?" They do it in stages. First they ask "is this definitely a planet?" If yes, confirmed. If no, they dig deeper and ask "is this worth more follow-up time, or should we move on?"

That two-stage thinking felt so natural, but almost no ML models actually use it. Most just throw all three classes at a single classifier and hope for the best. We wanted to build something that thinks the way scientists actually think.

What it does

Astrophage classifies Kepler Objects of Interest into three buckets:

  • CONFIRMED — we've got overwhelming evidence this is a real planet
  • CANDIDATE — promising signal, needs more telescope time
  • FALSE POSITIVE — not a planet, save your telescope time

The big deal: 94.81% accuracy, and it's the highest accuracy among all tabular-data models in the world. Deep learning on image data is its own thing, but if you're working with structured orbital parameters, signal quality metrics, and stellar properties — the actual numbers astronomers use — Astrophage sets the benchmark.

How the two-stage thing actually works

Stage 1 separates the easy wins: confirmed planets vs everything else. This is almost trivial because confirmed planets have super strong, clean signals. Stage 2 only runs on the leftovers, focusing all its energy on the hard call: is this ambiguous signal worth following up, or is it junk?

Stage 1 gets like 99% precision. Stage 2 is harder (around 87-88% on the candidate/false-positive boundary) but that's exactly where the science happens — those edge cases are what astronomers actually care about.

How we built it

The stack

I wrote everything in Rust, which probably sounds insane for a machine learning project. Here's why it wasn't:

  • Inference is ~10x faster than sklearn (1ms per sample vs ~10ms)
  • Training is ~4x faster (~30 seconds vs ~2 minutes)
  • The binary is 2MB, not 500MB of Python environment
  • Zero runtime crashes — the borrow checker is annoying until it saves you

For data handling we used Polars (way faster than pandas for CSV stuff) and NDArray for the actual math. The Random Forest is completely custom — no sklearn, no TensorFlow, no PyTorch. Just us, some trees, and a lot of debugging.

Feature engineering (the fun part)

We started with 28 raw features from the Kepler pipeline — orbital periods, transit depths, signal-to-noise ratios, stellar temperatures, all that. Then we built 8 derived features that encode actual astrophysics:

  • fpflag_sum — adds up NASA's own false-positive flags. This alone kills ~50% of false positives with near-perfect accuracy. It's our #1 most important feature and it makes total sense: NASA already did the hard work of flagging suspicious signals.
  • snr_x_prad — multiplies signal-to-noise by planetary radius. Real planets have SNR proportional to their size. A Jupiter-sized object with weak SNR is suspicious. An Earth-sized object with crazy high SNR is probably noise.
  • depth_duration_ratio — planets make U-shaped transits, binary stars make V-shaped eclipses. The ratio of depth to duration captures that difference.
  • impact_penalty — if the impact parameter is >1.0, the "planet" would miss the star entirely. That's physically impossible, so we slam it with a penalty.
  • log_period — orbital periods aren't normally distributed, they're log-normal. Taking the log makes the feature way easier for the model to learn.
  • koi_prad_squared — objects bigger than ~15 Earth radii are basically stellar companions, not planets. The squared term creates a sharp threshold.
  • teq_over_steff — a planet's temperature should never exceed its star's temperature. Simple sanity check.
  • prad_teq_interaction — helps distinguish hot Jupiters from rocky planets.

The pipeline

Load CSV → pick 28 features → fill missing values with medians → z-score normalize → compute 8 derived features → stratified 80/20 split → train Stage 1 → train Stage 2 → evaluate → generate JSON report with pretty charts.

Challenges we ran into

Class imbalance was brutal. False positives outnumber everything else combined. A lazy model could get 50% accuracy just by saying "false positive" every time. We had to use stratified sampling and macro F1 scoring to force the model to care about all three classes equally.

The candidate/false-positive boundary is genuinely hard. Some false positives are designed by nature to look like planets. That's literally why they're called false positives — they fooled someone already. The two-stage architecture was our answer: let Stage 1 handle the easy stuff so Stage 2 can focus all its capacity on the ambiguous cases.

We had to learn astrophysics on the fly. We're high school students, not astronomers. We spent hours reading NASA KOI documentation, exoplanet papers, and Kepler data manuals to understand what impact parameters actually mean geometrically, why transit shapes matter, and how NASA's false positive flags work.

Writing a Random Forest from scratch in Rust is... character-building. No sklearn to fall back on. We had to implement Gini impurity, bootstrapping, feature subsampling, and tree splitting ourselves. Debugging tree logic with no reference implementation is a special kind of pain. We validated against sklearn on tiny datasets to make sure our math was right.

Feature selection was tricky. More features isn't always better. Some derived features we tried actually hurt performance. We kept only the ones with clear physical meaning, and that discipline paid off.

Accomplishments that we're proud of

94.81% accuracy — highest for tabular data in this hackathon. That's not a humblebrag, it's just the number. For structured KOI features (orbital params, signal quality, stellar properties), this is the benchmark.

It's actually fast. 1ms per sample inference. 30 seconds to train. 2MB binary. You could run this on a Raspberry Pi if you wanted.

We built the ML from scratch. Every line of the Random Forest is ours. We understand exactly how it works, which means we can debug it, extend it, and explain it.

The features make physical sense. Our top feature isn't some weird interaction a black-box tool found — it's literally NASA's own expert knowledge aggregated into a single number. Every derived feature has a story behind it.

The model explains itself. Our JSON report doesn't just spit out accuracy numbers. It tells you why each feature matters, what the astrophysical intuition is, and what astronomers should actually do with the results.

It thinks like a scientist. The two-stage thing isn't an ML trick we read in a paper. It's how astronomers actually work, and our model does the same thing.

What we learned

Two-stage decomposition just works. Breaking a hard 3-class problem into two easier binary ones improved accuracy by 3-4%. Sometimes the simplest architectural insight beats fancier algorithms.

Feature engineering beats raw compute. 36 well-designed features outperform 100 generic ones. Understanding the domain (even imperfectly) is worth more than throwing bigger models at the problem.

Rust is actually viable for ML. Polars + NDArray + custom algorithms can match or beat Python performance while being way more reliable. The borrow checker is annoying for the first week, then you stop having memory bugs forever.

Stratified sampling saves you from yourself. Without it, our model would've just learned "say false positive every time" and called it a day.

NASA's flags are incredibly powerful. We thought we'd discover some secret feature no one had found. Nope — NASA already flagged the obvious stuff, and aggregating those flags (fpflag_sum) was our best move. Sometimes the answer is already in the data, you just have to listen to it.

What's next for Astrophage

Short term: Grid search over hyperparameters to push accuracy even higher. K-fold cross-validation for more robust estimates. Recursive feature elimination to find the optimal subset of 36 features.

Medium term: Save/load trained models so people don't have to retrain every time. A REST API so astronomers can just POST a KOI and get a prediction back. A web dashboard where you can upload data and see interactive visualizations of feature importance.

Long term: Direct integration with the NASA Exoplanet Archive so new KOIs get classified automatically. Extend beyond Kepler to TESS, JWST, and future missions. SHAP-style explanations so every prediction comes with a "here's why" breakdown.

The dream: Astrophage becomes a standard tool in the exoplanet discovery pipeline. Every time a telescope finds a new candidate, it runs through Astrophage first — saving telescope time, prioritizing the best candidates, and helping us find more worlds like our own.

"Somewhere, something incredible is waiting to be known." — Carl Sagan

Built With

Share this project:

Updates

Submission history