-
-
Project Main -- Page
-
Score 1
-
Score 2
Inspiration
When we look up at the night sky, we are no longer just looking at stars; we are looking at potential new worlds. The
launch of the Kepler Space Telescope revolutionized our understanding of the cosmos, providing terabytes of transit
light curve data. However, the bottleneck in exoplanet discovery is no longer data collection—it is data analysis.
Initially, we were inspired by Google Brain's 2018 paper where they used a 1D-Convolutional Neural Network (CNN) to
discover Kepler-90i. We wanted to build upon their foundation to classify planetary candidates. But as we dug deep into
the datasets provided by the NASA Exoplanet Archive (specifically the cumulative KOI table), we encountered a glaring
problem that plagues many beginner machine learning models in astronomy: Data Leakage.
We noticed that early, naive models could achieve 99% accuracy simply by looking at the koi_fpflag columns (flags
indicating if an object was a binary star or an instrument error). This wasn't artificial intelligence; it was just the
model memorizing answers that NASA scientists had already calculated. Our true inspiration for Kepler-AI became building
an "Honest Physics" classifier—a model that strips away all human-annotated cheat codes and forces the algorithm to
learn the actual underlying astrophysics of planetary transits to achieve a State-of-the-Art (SOTA) 96.87% accuracy.
## What it does
Kepler-AI is a robust, physics-informed machine learning pipeline that classifies exoplanetary candidates from the NASA Kepler dataset into two distinct categories: Real Planet or False Positive.
Instead of feeding raw, uncontextualized time-series data or human-annotated vetting flags into a black box, Kepler-AI
bridges the gap between astrophysics and data science. It ingests raw physical measurements (like orbital period,
stellar mass, planetary radius, and stellar temperature) and automatically engineers advanced astrophysical features
using fundamental laws of physics.
Once the data is transformed, a hyper-optimized XGBoost classifier evaluates the geometric and orbital properties of the
target star system. Finally, the model utilizes SHAP (SHapley Additive exPlanations) to ensure the AI's decision-making
process is completely transparent and grounded in actual science, breaking down exactly why a celestial object was
classified as a planet or a false alarm.
## How we built it
We built Kepler-AI through a rigorous, multi-step data science lifecycle:
Data Acquisition & Curation:
We utilized the NASA Exoplanet Archive's TAP (Table Access Protocol) API to programmatically download the confirmed
planetary_systems dataset and the cumulative Kepler Objects of Interest (KOI) dataset. We isolated the confirmed planets and strictly extracted the False Positives from the KOI table to construct a highly balanced, robust master dataset.Handling the Void (MICE Imputation):
Astronomical data is notoriously messy; telescopes cannot perfectly measure every parameter for every star due to
transit noise. Instead of dropping valuable data (which leads to data starvation), we utilized MICE (Multiple Imputation by Chained Equations). By running 10 iterations of MICE, we predicted the missing stellar masses and radii by leveraging the correlations between other available features (like stellar effective temperature,T
eff
).
- Physics-Informed Feature Engineering:
This was the core of our pipeline. We explicitly taught the model astrophysics by creating synthetic features derived
from fundamental formulas:
• Kepler's Third Law Proxy: We engineered a feature to check the consistency between the orbital period (P) and semi-
major axis (a) given the star's mass (M), based on:
4π²
P² = ───a³
GM
• **Stellar Density (
ρ
*
):** We calculated the approximate density of the host star to differentiate between main-sequence stars and giants
(which rarely produce clean planetary transits):
M
*
ρ = ──────
* 4 3
───πR
3 *
• Geometric Transit Probability (Pₜᵣₐₙₛᵢₜ): We calculated the likelihood of an observer on Earth actually seeing the
transit based on the star's radius and the planet's orbital distance:
R
*
P ≈ ────
transit a
- Model Architecture & Explainability:
After experimenting with Deep Neural Networks and TabNet, we settled on XGBoost. Tree-based ensembles excel at capturing non-linear thresholds in tabular data (e.g., the sharp boundary where a planet becomes too large and must be a brown
dwarf). We evaluated the model using rigorous train-test splitting and validated the predictions using a SHAP
TreeExplainer to guarantee the model wasn't relying on background noise.
## Challenges we ran into
The "Data Starvation" vs. "Overfitting" Dilemma:
Early in the project, we attempted to use a 1D-CNN directly on the raw transit light curves. However, because we
strictly refused to use data-leaked features, our CNN faced severe data starvation. There simply weren't enough cleanly
labeled, noise-free transits to prevent the deep learning model from memorizing the training set and failing on the
validation set.
The Data Leakage Trap:
When we transitioned to tabular data, our first XGBoost iteration hit 99.5% accuracy. We immediately knew something was
wrong. After conducting a feature importance analysis, we realized the model was relying entirely on koi_fpflag_nt and
koi_fpflag_ss (NASA's vetting flags). We had to completely purge our dataset of these columns, which plummeted our
accuracy to ~83%. It took days of intense physical feature engineering to rebuild the model's accuracy legitimately.
Corrupted API Downloads:
During our final testing phase, a cancelled API call left a corrupted, half-downloaded 250MB JSON file on our disk. This
caused our Jupyter Notebook to crash entirely due to parsing errors. We had to implement safer requests streaming with
chunking and file validation to ensure our data pipeline was bulletproof and reproducible.
## Accomplishments that we're proud of
• Shattering the Accuracy Ceiling: We are incredibly proud of rebuilding our model from an 83% baseline to a staggering
96.87% Final Accuracy (and an F1-Score of 0.972) using only raw geometric and physical features.
• True Reproducibility: We wrote a custom Python script that dynamically generates, executes, and bakes our final
Jupyter Notebook. A judge or scientist can clone our repository, run a single command, and perfectly replicate our 96.
87% accuracy from scratch.
• SHAP Transparency: By using SHAP, we proved that our model learned actual physics. Our plots show that the model
rightfully penalizes objects with excessively large radii (which are likely binary stars, not planets) and rewards
objects with high transit probabilities.
## What we learned
We learned that in scientific machine learning, domain knowledge is more powerful than raw compute. A simple XGBoost
model empowered by Kepler's Third Law and Stellar Density will consistently outperform a complex Deep Neural Network
that is blindly guessing on raw data. We also learned the hard lesson that if an accuracy score looks too good to be
true early in a project, it is almost certainly data leakage. Honest science requires rigorous auditing of your feature
space.
## What's next for Kepler-AI
While Kepler-AI currently analyzes tabular stellar and geometric properties, the next evolution is to create a true
Multi-Modal Ensemble.
We plan to reintegrate the time-series light curves by training a Vision Transformer (ViT) on the folded transit graphs,
and then stacking the Transformer's output with our current XGBoost tabular engine. Furthermore, we want to adapt
Kepler-AI to ingest data from the newer TESS (Transiting Exoplanet Survey Satellite) mission, allowing our "Honest
Physics" architecture to discover entirely new exoplanet candidates that humanity has never seen before.
Log in or sign up for Devpost to join the conversation.