The Martingale: Volatility Prediction Model

Executive Summary | Winter 2025

---

1. Architecture Overview

Core Model: XGBoost with Quantile Regression

I chose XGBoost as my primary model due to its proven performance on tabular financial data and its ability to handle feature interactions without extensive preprocessing. The key innovation was using Quantile Regression (70th percentile) instead of standard mean squared error optimization, which directly addresses the tendency of regression models to underpredict volatility spikes.

NLP Component: FinBERT + Sentiment Dynamics

For sentiment analysis of financial news headlines, I deployed FinBERT (ProsusAI/finbert) - a BERT model fine-tuned specifically on financial text. This provides significant advantages over simpler approaches like VADER:

• Domain-specific understanding: FinBERT understands financial terminology and context (e.g., "bearish", "bullish" vs generic negative sentiment)
• Confidence scoring: Beyond polarity (-1 to +1), I extract confidence scores indicating how decisive the sentiment is
• Shannon Entropy: I also compute text entropy as a proxy for information complexity/uncertainty in news since uncertainty is what drives volatility not extreme negative or extreme positive

Key Insight: Raw sentiment polarity has low predictive power for volatility because a persistently negative (or positive) market is priced in. What actually drives volatility is the switch — a sudden reversal in sentiment, a spike in uncertainty, or discourse/disagreement in the news. I engineer features that capture these dynamics:

NLP Feature	Signal	Rationale
Polarity_Delta1/3	Sentiment momentum	Rapid sentiment shift → volatility
Polarity_SignFlip	Regime reversal	Positive↔negative flip = market surprise
Polarity_Surprise	Deviation from norm	Sentiment far from 5-day average = shock
Discord	Entropy × (1 - \	polarity\
Entropy_Spike	Sudden entropy jump	New information/uncertainty entering market
NLP_x_Vol	Sentiment × technical vol	Sentiment shock amplified by volatile environment
Discord_x_RV	Discord × realized vol	Disagreement compounds fear during high-vol periods
Surprise_x_HMM	Surprise × HMM high-vol prob	Sentiment shock aligned with high-vol regime

Regime Detection: Hidden Markov Model (HMM) — Causal Inference

A 3-state Gaussian HMM identifies latent market regimes (low/medium/high volatility). This captures the well-documented volatility clustering phenomenon. Critically, I use causal (forward-only) inference rather than the standard forward-backward algorithm to prevent future data leakage — each day's regime probability depends only on past and current observations.

The HMM provides:

• State probabilities for each regime (forward-only)
• Regime entropy (uncertainty about current state)

Feature Engineering: 49 Total Features

Category	Count	Features	Purpose
NLP Base (4)	4	Polarity, Confidence, Length, Entropy	Raw sentiment & news complexity
NLP Dynamics (6)	6	Polarity MA3/MA5, Delta1/3, SignFlip, Surprise	Sentiment change & reversals
NLP Discord (5)	5	Discord, Entropy Delta/Spike, Confidence Delta/MA3	Disagreement & uncertainty signals
NLP Robustness (1)	1	HasHeadline	Graceful degradation when headlines disappear
NLP Interactions (3)	3	NLP×Vol, Discord×RV, Surprise×HMM	Cross-feature amplification
Volatility (11)	11	Std, MAD, Range, RV at 5/20/50 day windows	Multi-scale volatility
Tail Risk (5)	5	MaxAbs, TailCount (1%, 2%, 2σ)	Extreme event detection
Advanced Vol (3)	3	VolOfVol, Delta Vol, Downside Vol	Volatility dynamics
Distribution (4)	4	Skew, Kurtosis, Min/Max returns	Tail shape
Ratios (2)	2	Vol ratios across timeframes	Regime change signals
HMM (4)	4	State probabilities, entropy	Market regime

---

2. Risk Management Strategy

Problem: Conservative Predictions

The fundamental challenge in volatility prediction is that models trained with MSE loss converge to predicting the conditional mean, which systematically underestimates tail events. My diagnostics showed:

• Predictions had only 33% of actual variance
• High-volatility capture was only 37% (model captured barely 1/3 of spikes)

Solution: Multi-Layered Risk Management

1. Quantile Regression (Primary Defense) Instead of predicting E[Y|X], I predict the 70th percentile. This shifts the entire prediction distribution upward, reducing the penalty for overpredicting while maintaining responsiveness to volatility signals.

2. Sample Weighting (Aggressive Spike Emphasis) I apply asymmetric weights during training:

• Base samples: 1x weight
• Top 25% volatility: 3x weight
• Top 10% volatility: 6x weight
  
• Top 5% volatility: 12x weight

This forces the model to prioritize accuracy on high-volatility days.

3. Tail-Specific Features Features like Tech_TailCount_2pct_20 (count of >2% daily moves in last 20 days) and Tech_MaxAbs_20 directly signal elevated tail risk before it materializes in the target.

4. Time-Based Validation with Purge Gap I use strict chronological train/validation splits (80/20) with a 100-day purge gap between train and validation to prevent data leakage through the 100-lag features. No shuffling.

5. Extrapolation Scaling for Out-of-Range Events Since XGBoost cannot predict beyond training range and the private test likely contains extreme events (e.g., COVID crash), I apply post-prediction scaling when technical signals (RV, MaxAbs, Range) exceed the training P99. This uses a dampened sqrt scaling to prevent wild overestimates while allowing predictions to break through the training ceiling.

6. Full-Data Retraining Hyperparameters are tuned on the 80/20 purge-gap split, but the final submission model is retrained on 100% of training data to maximize signal.

---

3. Biggest Failure: The Conservative Bias Problem

Initial Approach (Failed)

My first iteration used standard XGBoost with MSE loss. Despite achieving decent RMSE, the predictions were visually "flat" - the model simply predicted values close to the historical mean, completely missing volatility spikes.

Why it failed:

• MSE penalizes over-predictions and under-predictions symmetrically
• Volatility spikes are rare (top 10% by definition)
• The model learned to "play it safe" by predicting moderate values
• Result: Good average accuracy, terrible tail performance

What I Tried (Partial Success)

Approach	Result	Issue
Log-transform target	Slight improvement	Still mean-reverting
Asymmetric loss	Better spike capture	Implementation complexity
Distribution scaling	Manual post-processing	Lost model signal
More features	Marginal gains	Didn't fix core bias

Final Solution

Quantile Regression was the breakthrough. By explicitly optimizing for a higher percentile, we fundamentally changed the model's objective from "minimize average error" to "predict where volatility will likely exceed."

Results after fix:

• Variance Ratio improved from 33% → ~55%
• High-Vol Capture improved from 37% → ~55%
• Model now tracks spikes rather than smoothing them away

Key Lesson

In volatility prediction, RMSE is a misleading metric. A model with higher RMSE but better spike capture is more useful for risk management than one optimizing pure accuracy. The business objective (capturing volatility spikes) must drive the loss function choice.

---

4. Summary

Component	Choice	Rationale
Base Model	XGBoost	Robust on tabular data, handles interactions
NLP	FinBERT	Domain-specific financial sentiment
Regime	3-state HMM	Captures volatility clustering
Loss	Quantile (70%)	Reduces conservative bias
Weighting	12x on extreme days	Forces spike attention
Validation	100-day purge gap	Honest evaluation, no lag leakage
Extrapolation	Sqrt-dampened scaling	Breaks training ceiling for crashes

The architecture balances sophistication (FinBERT, HMM with causal inference) with robustness (XGBoost), while the training strategy (quantile loss + aggressive sample weights + extrapolation scaling) directly addresses the core challenge of volatility prediction: capturing rare but important extreme events. The model retrains on 100% of training data for submission while maintaining honest evaluation via purge-gap validation. While more complex models such as LSTM were also considered and tested, given the small dataset and the competition's multiple-regime structure, LSTM would likely overfit the training data and fail to generalize across different regimes.

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for NLP+Marker+HMM -> XGBoost Strategy

Built With

• numpy
• pandas
• python
• sklearn
• torch
• xgboost