ExoVet

Separating real worlds from false alarms using the same physics NASA's own vetting pipeline relies on

Comment

What it does

ExoVet is a machine learning classifier that takes a Kepler Object of Interest's raw transit and stellar measurements and predicts whether it's a CONFIRMED planet, an unresolved CANDIDATE, or a FALSE POSITIVE — the same three-way call NASA's own vetting process makes, at 86.6% accuracy and 0.835 macro-F1 on held-out data.

Beyond the classifier itself, it ships as a complete, reusable pipeline: a saved model bundle that scores brand-new KOI data — a full CSV batch or a single hand-entered candidate — without retraining, and gracefully handles missing fields rather than crashing. It also includes a full SHAP explainability layer, so every prediction can be traced back to which measurements drove it, not just what the answer was.

How we built it

Cleaning first, modeling second. Beyond the koi_score column the challenge organizers had already removed, I found three more groups of columns that leak the target: koi_pdisposition (the pipeline's own disposition guess, matching the true label on 99.9% of rows), the four koi_fpflag_* columns, and kepler_name (only ever assigned after confirmation). All were dropped before any modeling — 40 columns removed in total.

Physics-grounded feature engineering. Rather than feeding in raw columns unchanged, I engineered features tied to actual transit physics. For example, comparing the observed transit depth to what a clean transit should produce given the planet-to-star radius ratio:

$$\text{expected depth (ppm)} = \left(\frac{R_p}{R_\star}\right)^2 \times 10^6$$

and checking stellar density consistency between the transit-fitted density and the catalog value:

$$\rho_{\text{catalog}} = \frac{M_\star}{R_\star^3}$$

A large mismatch between the two hints the transit signal may not actually belong to the star it's fitted around — a real false-positive signature.

Model selection, tested rather than assumed. I trained and compared Random Forest, XGBoost, and LightGBM, then tuned the best (XGBoost) via RandomizedSearchCV — optimizing for macro-F1 rather than accuracy, since accuracy alone is misleading with a 51%/29%/21% class split:

$$F1_{\text{macro}} = \frac{1}{3}\sum_{c \in {\text{CONF, CAND, FP}}} \frac{2 \cdot P_c \cdot R_c}{P_c + R_c}$$

I also built a full PyTorch neural network (MLP with BatchNorm, Dropout, early stopping on a genuine held-out validation set) to test whether deep learning would do better here, and a soft-voting ensemble of all three tree models.

Explainability with SHAP, to verify — not just assume — what the model had actually learned, down to per-prediction feature attribution rather than a single aggregate importance score.

Challenges we ran into

The leakage was the real challenge, not the modeling. Early runs hit ~99% accuracy, which felt great for about ten minutes before I realized it meant the model wasn't learning anything — it was just reading an answer key hidden in the feature columns. Finding and removing all three leakage sources, and accepting an honest 86.6% over a fake 99%, was the single most important decision in the whole project.

More complexity didn't mean a better model — and proving that took real work. I tried two more engineered features, LightGBM, a 3-model ensemble, and a from-scratch neural network, genuinely expecting at least one to beat the tuned XGBoost baseline. None did, meaningfully. Confirming that took actually running the experiments end to end rather than assuming they'd help — including re-tuning hyperparameters specifically for the new feature set to rule out that the new features were the problem, rather than stale hyperparameters.

CANDIDATE is a genuinely hard class, and understanding why mattered more than trying to brute-force past it. I worked through the actual math: even with perfect classification on the other two classes, CANDIDATE recall would need to reach 75.8% to hit 95% overall accuracy — and no model, including the neural net, got past 74%. That's not a failure of tuning; CANDIDATE is defined as a signal Kepler's own extensive pipeline and follow-up observations couldn't yet resolve either.

Accomplishments that we're proud of

  • Catching three separate target-leakage sources that would have quietly produced a meaningless "99% accurate" model — and being willing to report the real, lower, honest number instead.
  • An engineered feature (centroid-offset significance — offset divided by its own measurement uncertainty) breaking into the top-10 SHAP-ranked features, meaning the physics reasoning behind it actually mattered to the model, not just to me.
  • Watching SHAP confirm that the model's top features (photocenter offset measurements) are the same diagnostic real astronomers use to catch background-contaminated false positives — reconstructed from raw measurements, never shown the label that encodes it directly.
  • Building past the notebook: a real inference pipeline that scores brand-new KOI data — batch or single-candidate — without retraining, and degrades gracefully instead of crashing when fields are missing.
  • Testing the neural network and ensemble approaches properly instead of assuming either would obviously help or obviously fail — and reporting the negative result honestly.

What we learned

That accuracy alone lies on imbalanced data, and macro-F1 is what actually tells you whether a model is learning the hard classes or just the easy majority one. That more model complexity is not free performance — tree-based models, general research on datasets around this size, and my own from-scratch neural network experiment all agreed on that, for once. That the physics behind a feature matters as much as the number itself — engineering a significance ratio out of a raw measurement did more than just adding more raw columns would have. And that a negative result, reported honestly with the reasoning behind it, is worth more in a real analysis than a suspiciously perfect number with no explanation.

What's next for ExoVet

  • A hierarchical two-stage classifier: FALSE POSITIVE-vs-rest first, then CONFIRMED-vs-CANDIDATE on what's left — since CANDIDATE's difficulty may partly come from being forced to compete against both other classes at once in a single flat 3-way split.
  • SMOTE or targeted resampling specifically for the CANDIDATE class, to see if the recall gap closes without dragging precision down elsewhere.
  • Incorporating raw light-curve time series (not just the derived tabular summary statistics) via a proper 1D-CNN — published work on this exact dataset suggests this is where the next real accuracy gains are, beyond what tabular features alone can offer.
  • Cross-mission generalization to TESS data, to see whether the same engineered features and vetting logic transfer to a different telescope's systematics.
  • A lightweight public web demo around the existing inference pipeline, so anyone — not just someone comfortable with a Jupyter notebook — could paste in a candidate's measurements and get an explained prediction back.

Built With

Share this project:

Updates