XGBoosted Flash Flood Predictions (XGBFFP)

XGBFFP turns XGBoost flash-flood forecasts into interactive, explainable guidance, with its 75- and 100-km models capturing over 90% of moderate-or-greater cases.

Comment

XGBoosted Flash Flood Predictions

XGBoosted Flash Flood Predictions, or XGBFFP, is an experimental machine-learning decision-support platform for forecasting flash-flood potential from mesoscale convective systems. It combines real-time XGBoost predictions, official excessive-rainfall outlooks, environmental predictors, radar, flood alerts, observations, model verification, and explainability tools in one interactive website.

Inspiration

The predictions displayed by XGBFFP originated as an algorithm I developed for my Ph.D. dissertation. My dissertation research examined whether machine learning could improve forecasts of extreme rainfall and flash-flood potential associated with mesoscale convective systems, or MCSs.

After developing and evaluating the models, I wanted to take the project beyond static dissertation figures and research notebooks. A forecasting algorithm is much more useful when people can explore its predictions, compare them with operational guidance, inspect the factors influencing the model, and evaluate how it performs over time. I therefore extended the research into a website that generates and displays real-time predictions.

The motivation for this work is substantial. From 2014 through 2024, flash flooding caused approximately $6.4 billion in economic losses per year. During the warm season, an estimated 60%–80% of flash-flood events are associated with MCSs. Unfortunately, rainfall from these systems remains one of the most difficult meteorological phenomena to predict accurately.

MCSs frequently develop in environments characterized by relatively weak large-scale forcing. The smaller-scale processes that determine exactly where the systems form, move, and produce their heaviest rainfall are incompletely sampled by the observing network and may be poorly or only partially resolved by convection-allowing numerical weather prediction models.

These forecast errors matter because flood forecasts depend heavily on the placement and intensity of predicted rainfall. Even relatively small displacement errors can cause substantial errors in forecasts of streamflow and flooding. A storm producing the correct rainfall amount in the wrong watershed may appear meteorologically reasonable while producing a poor hydrologic forecast.

Gallus et al. (2025) analyzed object-based errors in quantitative precipitation forecasts from members of the High Resolution Ensemble Forecast system for MCS events occurring from 2018 through 2023. The largest source of error was the predicted location of the 24-hour precipitation footprint, followed by errors in precipitation intensity.

This is especially important in the central United States, where MCSs contribute roughly 30%–70% of warm-season rainfall. Improving these forecasts is therefore important not only for individual flash-flood events, but also for understanding a major portion of the region’s warm-season precipitation.

Building on my dissertation research

Earlier chapters of my dissertation explored whether machine learning could directly correct errors in numerical precipitation forecasts.

I trained multiple algorithms using object attributes calculated with the Method for Object-based Diagnostic Evaluation and near-storm environmental parameters. These algorithms predicted improved precipitation-centroid locations and precipitation intensities at five accumulation thresholds.

The best-performing model reduced displacement error by approximately 12% relative to the most accurate HREF member. Machine-learning adjustments also improved ensemble-mean precipitation intensity forecasts by approximately 30%–80%, depending on the threshold.

However, these improvements produced only limited gains in other object-based verification metrics and did not improve the Critical Success Index.

This revealed an important limitation. Correcting the center-of-mass centroid of an entire 24-hour precipitation footprint does not necessarily correct the placement of smaller, threshold-specific rainfall areas. At higher thresholds, precipitation objects become smaller, less continuous, and more sensitive to displacement. Similarly, correcting rainfall intensity changes the precipitation distribution but does not necessarily move the rainfall into the correct location.

These findings suggested that indirectly improving flash-flood forecasts through displacement and intensity corrections might be less effective than directly predicting the probability that rainfall will exceed a hydrologically meaningful threshold.

That became the basis for XGBFFP.

What XGBFFP predicts

XGBFFP uses four XGBoost models to predict the probability that observed 24-hour rainfall will exceed at least one form of Flash Flood Guidance within a neighborhood around each grid point.

Flash Flood Guidance estimates the amount of rainfall over a specified duration that would be required to cause small streams to overflow their banks. The target used by the models is whether observed rainfall exceeds at least one of the available 1-, 3-, or 6-hour guidance values.

Each radius represents a different tradeoff between spatial precision and event detection:

  • The 40-km model produces the most spatially conservative guidance.
  • The 60-km model captures more events while maintaining relatively limited false-risk issuance.
  • The 75-km model increases event detection while often retaining comparatively accurate risk placement and size.
  • The 100-km model produces the greatest overall detection and probabilistic skill, but can generate risk areas that are too large.

The models use environmental predictors from the Rapid Refresh model and gridded Flash Flood Guidance issued before the forecast period. Forecasts are valid from 12 UTC to 12 UTC the following day.

Unlike general-purpose excessive-rainfall models trained on every day of the year, XGBFFP was designed specifically for days involving MCSs. The training dataset contains 350 warm-season MCS cases from 2018 through 2023. Independent testing was performed using 45 MCS cases from June through August of 2024 and 2025.

How I built it

The underlying prediction system was developed in Python using XGBoost and a geospatial data-processing pipeline that combines meteorological, hydrologic, and observational datasets.

How the real-time workflow operates

The real-time system begins by retrieving the 12 UTC High-Resolution Rapid Refresh forecast. Before running the full machine-learning pipeline, it performs a simple MCS-detection check using two HRRR forecast fields:

  • Simulated satellite infrared brightness temperatures
  • Forecast precipitation rates

These fields are used to determine whether the HRRR predicts a sufficiently large, organized area of cold cloud tops accompanied by sufficiently intense rainfall. This check is not intended to provide a complete scientific classification of an MCS. Instead, it acts as a lightweight trigger that prevents the more computationally expensive forecasting workflow from running on days when the HRRR provides little indication that an MCS will occur.

If a potential MCS is detected, the normal XGBFFP workflow proceeds:

  1. Retrieve the 12 UTC HRRR forecast and perform the preliminary MCS-detection check.
  2. If an MCS is detected, retrieve the environmental model fields and Flash Flood Guidance required by the trained algorithms.
  3. Recreate the predictors and neighborhood statistics used during model training.
  4. Apply the trained 40-, 60-, 75-, and 100-km XGBoost configurations.
  5. Produce the gridded probability that observed 24-hour rainfall will exceed at least one 1-, 3-, or 6-hour Flash Flood Guidance value.
  6. Calculate the multi-configuration ensemble-mean probability.
  7. Generate the static forecast graphics and interactive map data.
  8. Publish the forecast and its valid period to the XGBFFP website and forecast archive.
  9. After the forecast period has ended, retrieve the available flood-proxy observations.
  10. Generate Practically Perfect fields and other verification products.
  11. Publish the event verification and update the running model-performance statistics.

This trigger-based approach allows XGBFFP to focus specifically on the type of warm-season convective events for which the models were developed. The ML models were trained on MCS cases rather than on all weather days, so checking for a forecast MCS before producing the guidance helps keep the real-time application consistent with the intended domain of the underlying research.

The interactive website allows users to compare:

  • The four XGBoost forecast configurations
  • The ML ensemble mean
  • The Weather Prediction Center Excessive Rainfall Outlook
  • Practically Perfect verification fields
  • Environmental predictor diagnostics
  • Radar imagery
  • NWS flood watches and warnings
  • Local storm reports
  • Flood-proxy observations used during verification

A location-briefing capability allows users to select a point and compare the forecast probabilities from the different ML configurations. This helps communicate model agreement, differences among neighborhood scales, and how the ML guidance compares with official outlooks.

The project also includes model-skill and explainability sections containing formal test-set results, rolling real-time verification, SHAP summaries, and dependence plots. These components are intended to make the model more transparent rather than presenting its predictions as an unexplained black box.

Model performance

The models were evaluated against several Unified Flooding Verification System flood proxies and against Practically Perfect risk fields derived from observed flooding information.

Across the 45 independent test cases, the 100-km model displayed the greatest overall skill among the four ML configurations and the WPC forecasts when the occurrence of any UFVS flood proxy was used as the reference. Excluding the 40-km configuration, the other ML models generally displayed skill comparable to or greater than the WPC forecasts.

The 100-km model had higher probabilities of detection than WPC at every evaluated threshold. At the greater-than-5% threshold, its average false-alarm ratio was comparable to that of WPC.

All ML configurations except the 40-km model had lower Brier Scores than the WPC forecasts, indicating greater probabilistic accuracy. The 100-km model had the lowest Brier Score overall. The largest improvement among adjacent ML configurations occurred between the 60- and 75-km targets. Lower values indicate better probabilistic accuracy.

Comparison with Practically Perfect

Practically Perfect fields are observation-based risk fields generated after an event. They are not forecasts. Instead, they provide an idealized spatial representation of where categorical risks would have been warranted based on the observed event.

All but the 40-km ML configuration had average Equitable Threat Scores comparable to WPC at the greater-than-5% threshold. At higher thresholds, the ML configurations generally had higher ETS values than WPC, except at the greater-than-70% threshold.

None of the forecast systems demonstrated skill at the greater-than-70% threshold. For the ML models, this was primarily associated with placement errors in the highest-risk areas. WPC did not issue any High Risks for the test cases.

Risk occurrence

The Practically Perfect fields contained at least a Slight Risk in all 45 test cases. WPC failed to issue a Slight-or-greater risk in seven of those cases.

The 60-, 75-, and 100-km ML configurations issued Slight-or-greater risks in all 45 cases. The 40-km configuration missed two.

The differences were more pronounced at the Moderate Risk threshold.

Practically Perfect contained a Moderate-or-greater risk in 36 of the 45 cases. WPC issued a Moderate-or-greater risk in only seven cases, capturing approximately 19% of the Practically Perfect Moderate-or-greater events.

The ML models captured substantially more:

  • The 40-km model captured 44% of Moderate-or-greater cases.
  • The 60-km model captured 61%.
  • The 75- and 100-km models captured more than 90%.

The increased detection from the larger-radius models involved a tradeoff. The 75- and 100-km configurations incorrectly issued Moderate-or-greater risks in approximately 21% of cases. In contrast, the 40- and 60-km models did not incorrectly issue any Moderate-or-greater risks.

This distinction is one of the reasons XGBFFP provides all four configurations instead of presenting one model as universally superior. The larger-radius products maximize detection, while the smaller-radius products provide more conservative guidance.

Overall, the 100-km model achieved the greatest aggregate skill but frequently produced Moderate and High Risk areas that were larger than those in Practically Perfect. The 60- and 75-km models often produced the most accurate balance of risk placement, size, detection, and false alarms. The 40-km model was the most conservative in both risk area and issuance frequency.

These results demonstrate why model performance cannot be summarized by a single score. A model can have better detection but larger forecast areas and more false alarms. Another model can produce fewer false alarms but miss a greater portion of significant events.

XGBFFP exposes these tradeoffs directly so users can compare the configurations rather than treating the model output as a single deterministic answer.

Challenges

One of the largest challenges was moving from a dissertation research workflow to a reliable real-time system.

The original algorithms were developed for retrospective analysis, where all required data were already available and individual cases could be inspected manually. A real-time application has to account for delayed data, incomplete files, changing source formats, missing observations, and differences between forecast and verification availability.

Another challenge was ensuring consistency between model training and real-time inference. XGBoost expects the same predictors, definitions, units, and preprocessing used during training. A real-time pipeline cannot simply fill missing scientific predictors with arbitrary values without potentially invalidating the prediction.

The workflow therefore required careful auditing of:

  • Predictor names
  • Data locations
  • Temporal accumulation periods
  • Neighborhood calculations
  • Flash Flood Guidance definitions
  • Grid alignment
  • Missing-data handling
  • Forecast valid periods
  • Verification timing

Publishing the forecasts introduced additional challenges. The website has to support both current and archived dates while allowing newer dates to contain features that older archived products may not have. The interface therefore needs backward-compatible data schemas and graceful handling of unavailable layers.

Verification also required careful separation of several different questions:

  • How well do the probabilities predict individual flood proxies?
  • How well do categorical ML risk areas overlap with Practically Perfect?
  • How frequently does each system issue Marginal, Slight, Moderate, and High risks?
  • How large are the risk areas?
  • How often are important events detected?
  • How frequently are risks issued when they are not supported by the observations?

Combining these questions into one number would conceal meaningful differences among the configurations. The website instead presents multiple complementary metrics, including ETS, probability of detection, false-alarm ratio, frequency bias, Brier Score, reliability, risk frequency, area, and placement.

What I learned

This project reinforced that building a skillful model and building a useful forecasting tool are different problems. Me being a meteorologist made it very easy to iterate and test the website to decide what features would be useful for making a product that is easily understood by users.

During model development, the emphasis is often on improving aggregate verification statistics. In an operational interface, users also need to know:

  • What valid period the forecast covers
  • Which model configuration they are viewing
  • How strongly the configurations agree
  • How the ML compares with official guidance
  • Which environmental factors influenced the forecast
  • Whether observations later supported the prediction
  • How much historical evidence supports the apparent performance

I also learned that greater event detection is not automatically better. The 75- and 100-km models captured more than 90% of the Moderate-or-greater cases, but they also issued more false risks and frequently forecast risk areas that were too large. The 60-km model was more conservative and missed more cases, but it avoided erroneous Moderate-or-greater issuance in the test set.

These tradeoffs are scientifically important and should be visible to users.

The project also demonstrated the value of explainability. SHAP plots and dependence plots do not prove physical causation, but they help identify the environmental variables the XGBoost models rely on and how changes in those predictors affect the forecasts. This provides an important layer of transparency when presenting machine-learning guidance alongside conventional meteorological products.

Finally, I learned how much engineering is required to move research into a continuously updated public product. Data acquisition, validation, reproducibility, deployment, error handling, documentation, and interface design became just as important as the model itself.

Role of Codex and GPT-5.6

I used Codex and GPT-5.6 to help convert the research code into a more complete public-facing application.

They assisted with:

  • Auditing the existing forecasting and publishing workflow
  • Organizing the research code into a real-time pipeline
  • Developing and refining the interactive website
  • Connecting forecast, verification, and observation layers
  • Adding location-specific forecast summaries
  • Building model-skill and explainability displays for the website
  • Improving compatibility between current and archived products
  • Identifying edge cases in missing-data and file-selection logic
  • Documenting the project and its scientific metrics

The meteorological definitions, training targets, predictor selection, verification methodology, model interpretation, and decisions about which tradeoffs were scientifically acceptable remained human-directed. Codex accelerated the implementation of this meteorologist-created product as a feature-rich real-time website.

What is next

Several extensions could improve XGBFFP.

The current models rely primarily on environmental information from the RAP. Future work could incorporate forecast ensembles rather than a single environmental model. Predictors from convection-allowing models or ensembles of convection-allowing models may provide additional information about convective organization, rainfall intensity, and uncertainty.

Operational numerical models also change over time. The planned retirement or replacement of systems such as the HREF creates challenges for machine-learning methods trained on historical model configurations. Transfer learning may provide a way to adapt relationships learned from one model system to another without rebuilding the entire framework from the beginning.

The models could also be updated incrementally as new verified MCS cases become available. This would expand the training dataset and allow evaluation of whether model performance changes as weather models, observing systems, and climate conditions evolve.

Other future improvements include:

  • Better calibration of Moderate and High Risk probabilities
  • Train models to predict Day 2 probabilities rather than just Day 1 (48 hours in the future)
  • Reduced westward displacement bias
  • Improved control of forecast risk area
  • Additional ensemble-based predictors
  • More detailed local SHAP explanations
  • Expanded seasonal and regional verification
  • Evaluation on non-MCS excessive-rainfall events
  • User feedback from operational forecasters and emergency managers

Impact

XGBFFP is not intended to replace official forecasts, watches, or warnings. It is experimental guidance designed to provide an additional perspective on days when an MCS may produce flash flooding.

The test results indicate that the ML guidance can draw attention to cases where higher risks may be warranted but are not represented in the official outlook. WPC captured only 19% of the Practically Perfect Moderate-or-greater cases in the test set, while the ML configurations captured between 44% and more than 90%, depending on the selected neighborhood radius.

By presenting multiple configurations, verification results, uncertainty, environmental context, and model explanations together, XGBFFP helps users understand both the potential value and the limitations of the predictions.

The broader goal is to demonstrate how dissertation research can be transformed into a transparent, testable, real-time decision-support tool for high-impact weather forecasting.

Built With

Share this project:

Updates