EWS

Inspiration

Current drought early warning systems classify risk at county scale. A pastoralist in Marsabit's rangeland and a farmer in its cropland see the same "stressed" label, even though their fields dry out at different rates and their drought responses differ. This spatial aggregation bias obscures field-level heterogeneity and delays targeted early action. We wanted to know: what if we forecasted at the native resolution of the satellite data itself, and were honest about where the model is uncertain?

What it does

A seven-stage pipeline ingests 30m Harmonized Landsat-Sentinel-2 imagery from Microsoft Planetary Computer, builds per-pixel NDMI and land-surface-temperature anomaly time series, and stratifies by ESA WorldCover land use before clustering drought-response regimes. Each pixel gets a Holt-Winters seasonal forecast and a two-stage trust gate (seapig global OOD + LoOP local outlier check) that classifies it as trusted, low-confidence, or withheld. An alert bot queries named locations and fires SMS alerts only on trusted pixels exceeding a -1.5σ NDMI anomaly threshold — never on uncertain pixels. The system knows what it doesn't know.

How we built it

• Stage 1: Year-by-year HLS streaming ingestion with consistent 30m grid alignment via stackstac and Icechunk versioned Zarr storage
• Stage 2: Fmask QA masking, Whittaker gap-filling, recency-weighted day-of-year climatology for NDMI and LST anomaly
• Stage 3: WorldCover-based stratification into 7 land-cover classes
• Stage 4: Per-stratum PCA embedding (2 components, 90% variance target) with full spatial coordinate tracking
• Stage 5: HDBSCAN temporal clustering with spatial subsampling — 23 clusters across 3 viable strata (cropland, rangeland, shrubland)
• Stage 6: Per-cluster Holt-Winters forecasting with 36-period seasonal decomposition
• Stage 7: Two-stage trust gating — deterministic classification, no black-box AI in the decision loop
• Alert bot: Icechunk query → deterministic threshold check → Africa's Talking SMS delivery

Stack: Python, xarray, scikit-learn, statsmodels, Icechunk, HDBSCAN, seapig, PyNomaly.

Challenges we ran into

• Cloud masking is brutal: 77% of HLS pixels flagged bad by Fmask. Whittaker smoothing and recency-weighted climatology were essential to recover usable signal.
• Clustering rediscovered land cover, not drought: Early runs without stratification grouped woodland vs. rangeland pixels rather than wet-year vs. dry-year trajectories. Stratification fixed this.
• A silent resampling bug collapsed every forecast: reindex to dekadal frequency did exact timestamp matching on nanosecond-precision satellite acquisition times, matching ~1 of 586 observations. Every cluster fell back to naive persistence. Switching to resample("10D").mean() fixed it.
• Shape mismatches between stages: xr.apply_ufunc silently reordered axes; raw numpy masks broadcast positionally and crashed. Wrapping masks as named xr.DataArray solved this.
• Trust gating was 686× too slow: Passing per-timestep embeddings (122k rows) instead of per-pixel embeddings (178 rows) to seapig caused 30+ minute hangs. Reshaping to per-pixel trajectories fixed both the speed and a downstream boolean-index crash.

Accomplishments that we're proud of

• Real Holt-Winters forecasts on 23 clusters with sensible AIC/SSE, not naive persistence — after fixing the dekadal resampling bug
• Deterministic trust gating that withholds 30–50% of pixels depending on stratum, without human-in-the-loop intervention
• End-to-end Icechunk versioning — every stage committed, every forecast traceable to its inputs
• An alert bot that honestly says "no signal here" instead of guessing on uncertain pixels
• Region-agnostic architecture — swap the bounding box in config.py and re-run

What we learned

The hard part isn't the model — it's knowing when not to predict. Our trust gate withholds alerts on half the landscape in some strata, not because the pipeline failed, but because the data doesn't support confident forecasting there. That's not a limitation to engineer around; it's the feature. A system that alerts everywhere is a system that alerts nowhere.

We also learned that "working" ML pipelines silently produce nonsense more often than they fail loudly. The dekadal resampling bug, the shape-mismatch crashes, the 686× slowdown — all were silent or nearly silent until we added targeted logging. Operational early warning needs observability, not just accuracy.

What's next for EWS

• Scale to county-level AOIs — the architecture supports it, runtime at full Marsabit County scale is the next bottleneck
• Swap Holt-Winters for a temporal transformer trained across clusters, using the trust-gated labels as training signal
• Validate against 2020–2023 Horn of Africa drought — qualitative comparison against FEWS NET IPC phase declarations

Built With

• climate
• clustering
• drought
• early-warning
• geospatial
• hdbscan
• hls
• icechunk
• pandas
• planetary-computer
• python
• remote-sensing
• satellite-imagery
• scikit-learn
• stac
• statsmodels
• time-series
• xarray
• zarr