Sigma Boi

Inspiration

I'd been following prediction markets for a while, but what really pulled me in was a simple question: if a contract trades at 64 cents, does it actually resolve YES 64% of the time? The answer, it turns out, is it depends — and that dependency is structured, domain-specific, and exploitable.I wanted to build a forecasting agent that didn't just parrot market prices back, but genuinely understood when to trust them, when to correct them, and how to combine market signals with external information intelligently.

What it does

Prophet is a multi-layer forecasting pipeline for Kalshi binary markets. Given a live market, it produces a calibrated probability estimate by combining: A market prior — the bid/ask midpoint as the base probability A sector-bucket calibration layer — empirical correction based on historical resolution data A fast order-flow model — a trained residual nudge based on public market patterns An LLM conviction layer — a gated language model that checks for verifiable, not-yet-priced public signals The final estimate is a weighted blend:p^=wmkt⋅pmid+wcal⋅pcal+wfast⋅Δfast+wllm⋅Δllm\hat{p} = w_{\text{mkt}} \cdot p_{\text{mid}} + w_{\text{cal}} \cdot p_{\text{cal}} + w_{\text{fast}} \cdot \Delta_{\text{fast}} + w_{\text{llm}} \cdot \Delta_{\text{llm}}p^=wmkt⋅pmid+wcal⋅pcal+wfast⋅Δfast+wllm⋅Δllmwhere each weight is dynamic, conditioned on data quality, confidence, and market context.

How we built it

The core insight came from going through historical Kalshi data and asking: for markets priced at xx x cents, what fraction actually resolved YES? I built a lookup table indexed by sector | subtype | days_before_close | price_bucket. For each cell I computed:

nn n — number of resolved contracts pˉmkt\bar{p}{\text{mkt}} pˉmkt — mean market odds at observation time p^emp\hat{p}{\text{emp}} p^emp — empirical resolution rate edge=p^emp−pˉmkt\text{edge} = \hat{p}{\text{emp}} - \bar{p}{\text{mkt}} edge=p^emp−pˉmkt

The results were striking. For the weather sector, markets priced at 46–50 cents resolved YES only 16.7% of the time — a gap of nearly −33 percentage points. For crypto price markets, the opposite: a 46–50 cent contract resolved YES 75% of the time (+25pp). Sports markets were much tighter, consistent with the academic literature on short-horizon calibration. I compared these findings against Le (2026), which analyzed 292 million trades across 327,000 contracts on Kalshi and Polymarket

Challenges we ran into

Thin data at the extremes. Cells with n<30n < 30 n<30 produce noisy empirical rates. A 100% empirical YES rate in a 91–95 cent bucket with n=7n = 7 n=7 tells you almost nothing. Smoothing and capping helped but didn't fully solve it. The LLM doesn't know what it doesn't know. Getting the conviction layer to correctly assess not_priced — whether a piece of information is already reflected in the market — was genuinely hard. The LLM is good at identifying what is true; it's poor at identifying whether traders already know it. Template misclassification. Routing "Will the world pass 2°C before 2050?" to a WEATHER template felt intuitive but was wrong — climate science questions have completely different calibration characteristics than daily weather markets. Weight calibration is circular. Choosing wcal,wfast,wllmw_{\text{cal}}, w_{\text{fast}}, w_{\text{llm}} wcal,wfast,wllm optimally requires historical forecasts to evaluate against — which I don't have yet. The current weights are principled estimates, not learned parameters.