VolEdge

Project Story

Regime-Adaptive Volatility Forecasting with Mixture-of-Experts

Financial markets are deceptively simple at first glance. A price goes up, a price goes down, and somewhere inside that movement lies information. What fascinated me about this competition was not predicting direction. That’s noisy and fragile. It was about predicting magnitude. The target was the absolute intraday movement:

$$ \text{Target} = 100 \times \left| \frac{\text{Close} - \text{Open}}{\text{Open}} \right| $$

In other words: How volatile will today be?

That framing changed everything. Instead of asking “Will the market go up?”, the question became “How turbulent will it be?” — a far more structural, regime-driven problem.

⸻

What Inspired the Approach

Early exploratory work showed something critical: volatility clusters. Large moves follow large moves. Calm periods persist. Crisis periods behave fundamentally differently from bull markets.

This is not just intuition — it’s documented extensively in financial econometrics. The HAR (Heterogeneous Autoregressive) model suggests that volatility is composed of multiple time horizons: • Daily component • Weekly component • Monthly component

This inspired a shift from naive rolling statistics to structured realised volatility features:

$$ \text{RV}k = \sum{i=1}^{k} r_i^2 $$

and its stabilised version:

$$ \log(\text{RV}_k) $$

I also incorporated an exponentially weighted moving variance (EWMA), inspired by RiskMetrics:

$$ \sigma_t^2 = \lambda \sigma_{t-1}^2 + (1 - \lambda) r_t^2 $$

This captures persistence — volatility does not forget instantly.

But even with strong features, something interesting emerged.

⸻

The First Real Insight: One Model Is Not Enough

My initial baseline used: • Ridge regression • Random Forest • LightGBM

Combined into an ensemble.

It performed well. But when I evaluated performance by regime — especially high-volatility slices — a pattern appeared: • The ensemble performed best during calm markets. • A HAR-enhanced Random Forest performed better during crisis regimes.

No single model dominated everywhere.

That was the turning point.

Instead of tuning endlessly, I reframed the problem:

What if volatility itself is regime-dependent — and the model should be too?

⸻

Building the Mixture-of-Experts

This led to a Mixture-of-Experts (MoE) architecture.

Two experts: • Calm expert: Baseline ensemble (stable, low-variance) • Crisis expert: HAR-enhanced Random Forest (tail-aware)

The final prediction became:

$$ \hat{y} = (1 - p(x)) \cdot \hat{y}{\text{calm}} + p(x) \cdot \hat{y}{\text{crisis}} $$

where p(x) is learned by a logistic regression gate using only price-derived regime features.

To train the gate without leakage, I used out-of-fold (OOF) errors to define when each expert performed better. The gate effectively learned:

Under which volatility structure does the crisis model outperform the calm model?

This was validated using expanding-window time-series cross-validation — a critical safeguard against lookahead bias.

⸻

The Engineering Process

The full pipeline evolved into layered stages: 1. Data ingestion and validation • Time ordering enforced • Schema verified • No leakage tolerated 2. Feature engineering • Rolling volatility • HAR multi-horizon realised variance • EWMA volatility • Volatility-of-volatility • Asymmetry (leverage effect proxies) 3. Model training • Ridge (low-variance linear bias) • Random Forest (nonlinear thresholding) • LightGBM (gradient boosting interactions) 4. Validation • 5-fold expanding window CV • Tail slice analysis (High95 volatility days) • Hard-regime fold inspection 5. Regime-adaptive blending (MoE) • Logistic gate trained only on OOF comparisons • Final adaptive weighted predictions

The most important lesson here was structural discipline. Financial time series are not i.i.d.; shuffling would have invalidated the experiment.

⸻

Challenges Faced

1. Distribution Shift

The training period (2009–2012) did not perfectly match the public leaderboard period (2013–2020). A model that improved crisis metrics in cross-validation did not always improve public performance — because the public slice was calmer.

This forced me to think probabilistically about private leaderboard weighting rather than chase public improvements.

⸻

1. Leakage Risk

The competition provided the full test dataset without targets. That introduces temptation — and risk.

Ensuring that: • No test-derived signals leaked into training • The gate used only OOF predictions • All splits were strictly forward in time

was critical.

⸻

1. Overfitting via Complexity

It’s easy to add more models, deeper trees, larger feature sets. But each added degree of freedom increases fragility.

The gate was deliberately kept simple: logistic regression. Low capacity. Transparent. Stable.

⸻

What I Learned

1.  Volatility is structural, not random.
2.  Multi-horizon components matter.
3.  Model diversity beats hyperparameter tuning.
4.  Regime-awareness is essential — markets are nonstationary.
5.  Validation discipline is everything.
6.  Blending can be principled — Mixture-of-Experts is a mathematically grounded response to nonstationarity.

⸻

Final Architecture Summary

The final system can be summarised as:

$$ \text{Volatility Features} \rightarrow \text{Expert Models} \rightarrow \text{Regime Gate} \rightarrow \text{Adaptive Prediction} $$

It began as a simple ensemble. It evolved into a regime-adaptive forecasting system grounded in financial econometrics and machine learning theory.

⸻

Closing Reflection

What began as a competition submission became an exploration of how markets behave under stress and calm.

The most important transformation was not technical — it was conceptual.

I stopped asking, “Which model is best?” and started asking, “Under which conditions is each model best?”

That shift — from global optimisation to conditional optimisation — changed the entire trajectory of the project.

And that is the lesson I will carry forward.

Built With

• python