Exoplanet Classification: Sorting Signal From Noise

output image

What it does

Classifies all 9,564 Kepler Objects of Interest from the NASA Exoplanet Archive into three classes: confirmed planet, candidate, or false positive (eclipsing binaries, instrument noise). SHAP explains every prediction in terms an astronomer would recognize: transit geometry, signal repeatability, centroid offset — all learned from transit-physics and stellar-parameter features alone.

Metric	Score
Accuracy	85.9%
Macro F1	0.830
F1 — False positive	0.907
F1 — Confirmed	0.903
F1 — Candidate	0.682

Inspiration

Kepler flagged thousands of periodic dimming events between 2009 and 2018, but a computer catching a dip in starlight isn't the same as confirming a planet. Vetting each signal by hand takes real astronomer time, and the archive still carries plenty of KOIs stuck in CANDIDATE limbo. So this project mirrors what vetting teams do: check the transit shape, check whether the signal repeats, check whether the dimming is happening on the target star at all.

How I built it

Two notebooks backed by four modules in src/.

01_eda.ipynb — audit the raw 9,564 × 140 table first.

19 entirely empty columns (eccentricity errors, stellar age, model fit stats) dropped.
Leakage hunted down: koi_score already absent (it is the vetting pipeline's own confidence score), koi_pdisposition near-copies the target, and kepler_name is only assigned to confirmed planets, so even its missingness leaks the label. The four koi_fpflag_* columns (not transit-like, stellar eclipse, centroid offset, ephemeris match) are outputs of NASA's automated false-positive vetting pipeline — they encode the target classification and are excluded as data leakage. All excluded, along with vetting comments, vetting status metadata, and data-validation report links.
KOI uncertainties come in symmetric ± pairs; _err2 columns are perfectly anti-correlated with their _err1 twin, so the mirrored halves were pruned.

02_modeling.ipynb — preprocessing, features, training.

Remaining missingness (≤16% per column) is median-imputed, medians fit on the training split only; the physical quantities are heavily right-skewed, so a mean would be dominated by the tails.

Eight domain-derived features, each tied to a piece of vetting physics:

Feature	Formula	Why
Depth per hour	depth / duration	Stellar eclipses deep and sharp; planetary transits shallow and gradual
Radius ratio	\(R_p / (109.2\,R_\star)\)	Ratio near 1 means the "planet" is a star \((1\,R_\odot \approx 109.2\,R_\oplus)\)
Event statistic ratio	MES / SES\(_{\max}\)	Real transits accumulate evidence across orbits; one-off glitches don't
Duration check	observed vs. \(P^{1/3}\)	Kepler's third law; large deviations flag grazing eclipses or blends
Log transforms	\(\log_{10}\) of period, depth, insolation	These span several orders of magnitude
SNR per transit	SNR / \(N_{\mathrm{transits}}\)	Strong-because-deep vs. strong-because-seen-many-times

Three models compared, stratified 5-fold CV on an 80/20 stratified split, seed 42:

Model	Accuracy	Macro F1
Logistic regression (baseline)	0.869	0.776
Random forest	0.884	0.821
XGBoost	0.859	0.830

A 20-configuration randomized hyperparameter search moved CV macro F1 only slightly and did not improve the held-out test score, so the near-default configuration is what gets reported rather than a cherry-picked run.

Challenges I ran into

Class imbalance. 51% false positive, 29% confirmed, 21% candidate. Optimizing for accuracy alone would have flattered a model that mostly calls things "false positive." Macro F1 became the primary metric, paired with balanced class weighting instead of synthetic oversampling: SMOTE-style interpolation on heavy-tailed physical quantities like orbital period can fabricate physically impossible objects, while weighting keeps every real observation.

How much credit to give the model. The four koi_fpflag_* columns come from Kepler's automated vetting pipeline. They are not legitimate features — they encode the very false-positive classification we are trying to predict. Using them as features would be textbook data leakage, and indeed training with them artificially inflated macro F1 to 0.922. We exclude them entirely: the honest model, trained on transit-physics and stellar-parameter features alone, achieves 0.830 macro F1 with most error concentrated on the CONFIRMED/CANDIDATE boundary where it physically belongs.

Accomplishments I'm proud of

Almost all remaining errors sit on the boundary between CONFIRMED and CANDIDATE, which is the correct place for a model to be uncertain: a candidate is, by NASA's own definition, a signal that hasn't cleared the confirmation bar yet, so overlap there reflects real physics rather than a modeling flaw.

SHAP backs this up. The top features are:

the multiple-event statistic (does the signal repeat?)
the centroid offset koi_dicco_msky (is the dimming on the target star, or a neighbor blended into the same pixel?)
transit signal-to-noise
planet radius — objects larger than about \(2\,R_{\mathrm{Jup}}\) are almost always small stars

That ranking tracks how astronomers vet these signals by hand: check the geometry, check the repeatability, check the signal quality.

What I learned

The dataset guide's decision to strip koi_score mattered more than expected: it forced actual feature engineering instead of letting one precomputed column do the whole job. More importantly, rigorous leakage control — excluding the koi_fpflag_* vetting pipeline outputs — kept the benchmark honest. The gap between the original inflated 0.922 and the honest 0.830 is a vivid lesson in how easily a "good" model can be a bad model wearing a training-contaminated score. Class-imbalance decisions mattered more than model choice; logistic regression to XGBoost moved macro F1 from 0.776 to 0.830.

What's next

Test on TESS candidates to check generalization past Kepler's instrument characteristics.
Lightweight web demo: predictions plus SHAP explanations for individual KOIs.
Per-feature ablation to isolate what each of the 8 engineered features contributes.