Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for Hindsight Alpha
Inspiration
Backtested trading strategies routinely look good in a demo and fall apart the moment they see genuinely new data, because the "winning" parameter was quietly picked using information that wouldn't have been available at decision time — classic look-ahead bias, usually only caught after the money is gone. Rather than building one more strategy and hoping it holds up, the question became: what if the agent itself refused to trade the instant its own selection process showed signs of that leak, in real time, against real bars — not in a one-off backtest chart?
What it does
Each run evaluates a small universe of liquid, optionable ETFs (SPY, GLD, XLK, XLV). For every symbol:
- Fetches real daily bars via Alpaca's Market Data API.
- Sweeps five candidate historical-volatility windows (10/20/30/60/90 days) for one rule: buy optionality when realized volatility is cheap relative to its own trailing distribution.
- Runs
hindsight_guard.check_selection_leakage— scores every candidate twice, once on the full history and once on everything except the most recent 20 bars, and compares the two winners. - Refuses to trade if the two winners disagree, if nothing clears the in-sample Sharpe bar, or if any candidate can't be scored at all — and prints why.
- Otherwise checks today's volatility regime with the vetted window; if cheap, a momentum tiebreaker picks a direction and the agent buys a near-the-money option contract, 7–21 days out, never sold short.
- Passes every risk gate (per-trade cap, total-exposure cap, per-sector cap, drawdown lock, consecutive-loss breaker) or doesn't trade.
Live result: XLK is refused on every single run against real bars — its two scoring passes disagree by a small but real margin (0.024 of a Sharpe unit). SPY, GLD, and XLV clear the check and have real paper orders behind them.
Public, live dashboard: every run, every refusal, every real order, republished every 30 minutes during market hours — https://s-papy.github.io/hindsight-alpha
How honest is the check itself?
A refusal mechanism whose own error rate is unknown is worth less than one whose error rate is measured — so it was measured twice, by two different constructions, and both numbers are published rather than only the flattering one: 22.6% false alarms (synthetic anomaly injection, varying the holdout) and 30.2% false alarms (score-level test against known ground truth). A third measurement, against the project's own interest: on a selection with no real edge and no leak — a winner picked by pure noise — the guard still certifies it 52% of the time. It catches look-ahead leakage; it does not catch overfitting without leakage, and nothing else in the pipeline does either.
How we built it
Python 3.9+, the official Alpaca CLI wrapped via subprocess, zero UI framework for the dashboard — one static HTML page served from GitHub Pages, fed by a JSON snapshot written by a script that runs with real credentials locally and never lets the public page touch Alpaca directly. Five launchd schedules (entry decision, exit monitor every 15 minutes, a caffeinate job holding the machine awake through market hours, and two dashboard-publish jobs) are versioned in the repo rather than living only on one machine. 700+ offline tests, standard library only, no network calls, run in CI on every push.
Challenges we ran into
Getting the leakage check itself trustworthy rather than decorative was the hardest part. An earlier version would have passed on a selection with real leakage undetected, close to the deciding margin later found on XLK (0.024 of a Sharpe unit, on two series overlapping 97% by construction). A second, unrelated challenge: Alpaca has no bracket or OCO orders on options, so the exit monitor polling loop is the only thing protecting an open position — it failed silently eleven times early on because the host machine slept through a scheduled maintenance window too short for Wi-Fi to reconnect. Diagnosed by matching failure timestamps to the system sleep log, fixed with a dedicated caffeinate schedule.
Accomplishments we're proud of
A real paper options order, submitted through the real Alpaca API and later closed by a real take-profit/stop-loss check — not a backtest chart. And publishing the guard's own false-alarm rate rather than only the flattering cases, including the 52% figure that argues against the project's own thesis on a specific edge case.
What we learned
That a check is only as credible as its own measured error rate, and that "it caught something" is a much weaker claim than "here's how often it's wrong, measured two independent ways, published either way."
What's next
Extending the leakage check to catch overfitting-without-look-ahead — the one failure mode the current guard explicitly does not catch, per its own 52%-false-certify measurement on noise-only selections.
Log in or sign up for Devpost to join the conversation.