Martingale Volatility Engine- HAR + LightGBM + Text Ensemble

• 

  architecture diag

Inspiration

Volatility forecasting is the backbone of options pricing, risk management, and systematic trading. We wanted to build something that would work in production — not just on a clean Kaggle leaderboard — meaning it had to be leakage-free, regime-robust, and reproducible under crash conditions.

What We Built

A three-model ensemble volatility forecasting pipeline for predicting the next-session absolute % move of SPY:

$$\text{Target} = 100 \times \left|\frac{\text{Close} - \text{Open}}{\text{Open}}\right|$$

The system uses only the provided dataset: 100 return lags, top-20 pre-market headlines, and Date.

Architecture

Raw Data (lags + headlines + date)
            |
            v
+--------------------------------------------------+
|              FEATURE STORE                       |
|  TF-IDF 3k + SVD-20 | HAR-RV d/w/m/q | Stats   |
|  text -> 20 dims     | sqrt(RV) ratios | Tails   |
+--------------------------------------------------+
       |                    |                |
       v                    v                v
+-----------+        +-----------+     +-----------+
|  MODEL A  |        |  MODEL B  |     |  MODEL C  |
| LightGBM  |        | LightGBM  |     |   Ridge   |
| 211 feats |        | HAR+lags  |     | 6 HAR feat|
|  seed=42  |        | seed=2025 |     |  a=1.0    |
+-----------+        +-----------+     +-----------+
       |                    |                |
       +--------------------+----------------+
                            |
                            v
          OOF Blender (grid search WA + WB + WC)
          4-fold TimeSeriesSplit -- 804 CV rows
                            |
                            v
            sqrt-space -> square inverse
                            |
                            v
                      submission.csv

Key Design Decisions

1. sqrt-space modeling We predict \( \hat{y} = \sqrt{\text{Target}} \) and invert via \( \max(\hat{y}, 0)^2 \). This compresses the heavy right tail of volatility, stabilizes RMSE gradients, and enforces non-negativity without artificial clipping.

2. HAR feature family Inspired by the Heterogeneous Autoregressive model for Realized Volatility (Corsi, 2009), we construct:

• \( \text{RV}_d = r_1^2 \) — daily realized variance
• \( \text{RV}w = \frac{1}{5}\sum{i=1}^{5} r_i^2 \) — weekly
• \( \text{RV}m = \frac{1}{22}\sum{i=1}^{22} r_i^2 \) — monthly
• \( \text{RV}q = \frac{1}{66}\sum{i=1}^{66} r_i^2 \) — quarterly

These capture volatility clustering across multiple time horizons simultaneously.

3. Tail frequency features Features like (|r_i| > 0.02).mean() over rolling windows detect when the market has been in a high-volatility regime — acting as a nonparametric regime indicator without requiring explicit regime labeling.

4. Three-model ensemble with Ridge anchor

• Model A (LightGBM, full features): captures nonlinear interactions across all 211 features
• Model B (LightGBM, HAR+lags): isolates the pure volatility signal without text noise
• Model C (Ridge, 6 HAR features): linear structural anchor — cannot overfit, degrades gracefully under distribution shift

5. Leakage-free validation TimeSeriesSplit(n_splits=4) ensures every validation fold is strictly in the future relative to its training fold. OOF RMSE is computed only on the 804 rows actually filled by CV folds — the first 202 rows (never in any validation set) are excluded from scoring.

Results

Public leaderboard score: 0.15002

Challenges

The single hardest bug: TimeSeriesSplit leaves the first 20% of training rows unfilled in OOF arrays. Those rows stay at zero, and inverse(0) = 0 — far from true targets — inflating reported RMSE by ~47%. We ran three full training cycles before catching this via a debug cell. The fix: a filled_mask built before any metric computation. We now assert oofA_corrected < 0.545 before every submission.

What We Learned

• HAR-family features are remarkably powerful for equity volatility with very small datasets
• Ensemble diversity matters more than ensemble size — three well-differentiated models beat five similar ones
• In time-series ML, the most dangerous bugs are silent: wrong OOF scoring looks plausible until you trace array shapes
• The Ridge anchor is underrated — it costs nothing and provides genuine robustness in regime shifts

What's Next

• GARCH-family residual correction as a fourth ensemble member
• Attention-weighted headline aggregation (lightweight, not BERT)
• Walk-forward expanding window retraining for live deployment

Built With

• kaggle
• lightgbm
• notebooks
• numpy
• pandas
• python
• ridge
• scikit-learn
• tf-idf
• timeseriessplit
• truncatedsvd