Kepler.ai: Physics-Informed Exoplanet Engine

Inspiration When exploring machine learning solutions for exoplanet discovery, we noticed a massive, recurring flaw: many amateur models applied to the NASA Kepler dataset boast a 99% accuracy rate. However, when we investigated the data, we discovered this was an illusion caused by Target Leakage. These models were utilizing the koi_fpflag_* columns flags assigned by human scientists after a candidate was vetted. The AI wasn't learning astrophysics; it was simply reading the scientists' notes.

Our inspiration was to build a scientifically rigorous, enterprise-grade machine learning pipeline that explicitly eradicates this data leakage. We wanted to create an engine that learns to identify exoplanets based purely on physical geometry, transit photometry, and stellar parameters, proving that an AI can understand the physics of the universe when guided by human domain knowledge.

What it does Kepler.ai is an automated, end-to-end classification engine that determines whether a Kepler Object of Interest (KOI) is a Confirmed Planet or a False Positive. Beyond simple binary classification, the pipeline:

Automatically Ingests & Validates: Pulls the latest cumulative data directly from the NASA/Caltech Exoplanet Archive and runs it through strict pandera schema validation to prevent silent pipeline failures.

Eradicates Leakage: Aggressively drops human-vetted flags to force the model to learn raw photometry.

Engineers Physics: Calculates complex physical realities (e.g., Planetary-to-Stellar Radius Ratios, Transit Orbit Fractions) to give the model mathematical shortcuts to the underlying astrophysics.

Explains its Reasoning: Utilizes an interactive Streamlit dashboard and SHAP (SHapley Additive exPlanations) to crack open the "black box," generating waterfall plots that explain exactly why a specific signal was classified as a planet or a fake.

How we built it We constructed Kepler.ai using a modular, enterprise-standard architecture rather than a single messy notebook.

Data Processing: We used pandas and numpy for vectorized mathematical operations, and pandera to enforce strict data contracts.

Modeling: We utilized LightGBM and XGBoost, configured via YAML files and an orchestrating train.py script. We used Optuna to tune hyperparameters, optimizing specifically for Macro F1 Score to balance the severe class imbalance in astronomical data.

Explainability & Evaluation: We integrated shap to map global and local feature attributions, and scikit-learn to generate rigorous evaluation metrics (ROC-AUC, Precision-Recall, and Calibration Curves).

UI: We wrapped the inference engine in a real-time streamlit dashboard.

Challenges we ran into The Target Leakage Trap: The biggest initial challenge was realizing our early models were cheating. Dropping the leakage columns plummeted our initial accuracy to ~85%. We had to rebuild the model's predictive power the hard way through rigorous feature engineering.

Strict Data Typing: Tree-based models like LightGBM crash when encountering raw string objects or un-encoded categorical variables. We had to build bulletproof preprocessing scripts to dynamically identify and drop or encode non-numeric NASA metadata.

The "Eclipsing Binary" Boundary Case: During our Error Analysis phase, we noticed our model was highly confident about certain False Positives. Using SHAP, we realized massive Eclipsing Binaries (stars orbiting stars) mimic the transit geometry of giant planets, tricking our geometric features.

Accomplishments that we're proud of The Ablation Study: We mathematically proved the value of our astrophysics domain knowledge. By training a baseline model on raw data, and a second model on our engineered features, we demonstrated a quantitative leap in Macro F1 and ROC-AUC scores. We proved our engineered features work.

Perfect Calibration: We achieved a Brier Score of 0.038. Our Reliability Diagram (Calibration Curve) tracks the perfect diagonal line, meaning when Kepler.ai says it is 80% confident a candidate is a planet, it is genuinely right 80% of the time. It is highly trustworthy.

Enterprise Architecture: Rather than a single Jupyter notebook, we built a fully reproducible pipeline triggered by a Makefile with YAML configuration files mirroring how ML is deployed in the real tech industry.

What we learned

We learned that in machine learning, context is everything. You cannot just throw an advanced algorithm like XGBoost at a CSV file and expect scientific breakthroughs. We had to study the Stefan-Boltzmann law, understand stellar geometry, and learn how to translate human domain knowledge into vectorized features that an AI can digest. We also learned how vital it is to use tools like SHAP to audit your model and keep it honest.

What's next for Kepler.ai: Physics-Informed Exoplanet Engine

The natural next step is to expand the engine beyond the retired Kepler mission. We want to adapt the data ingestion and validation schemas to process live transit data from TESS (Transiting Exoplanet Survey Satellite). Furthermore, to solve the "Eclipsing Binary" weakness identified in our error analysis, we plan to engineer specific features designed to detect secondary eclipses, pushing our False Positive rate even closer to zero.

Built With

• lightgbm
• numpy
• optuna
• pandas
• pandera
• pyyaml
• scikit-learn
• shap
• streamlit
• xgboost