Stormproof

Inspiration

I work in water resources and flooding. In every big storm the same thing happens: the power goes out, and the utility's outage map goes down with it. During Hurricane Beryl in July 2024, CenterPoint Energy's outage tracker in Houston was offline, and people tracked the power using the Whataburger app. BC Hydro's site failed in a 2015 windstorm at 18 times its normal traffic. PSEG Long Island upgraded its outage system just before Tropical Storm Isaias, and its map under-reported the outage.

The storms are forecast days ahead. The software teams just don't act on the forecast. Stormproof connects the two: weather warnings drive the post-code DevSecOps lifecycle.

What it does

Stormproof watches National Weather Service (NWS) alerts for a service area and sets a readiness level:

• Green: normal day.
• Amber (for example a flood warning or hurricane watch): load-test the public service at the expected storm traffic and scale it up.
• Red (hurricane, storm surge, tornado, flash flood warnings and more): block production deploys until the storm has passed, with a hazard-specific cooldown, because outage-map traffic peaks during restoration.

The demo protects the outage map of Bayou Power, a fictional Houston utility, running on Google Cloud Run. Replaying Hurricane Beryl from the NWS archive at 139 user actions per second:

Two GitLab Duo Agent Platform custom flows run the storm in GitLab:

• Storm Steward (triggered by pipeline events): opens and updates a Storm Watch issue with the readiness numbers, reviews every open merge request, clears storm-safe changes, holds risky ones as Draft during a freeze with the reason and the emergency route, and releases its holds and posts the all-clear when the storm ends.
• Storm Postmortem (triggered by a mention after a person closes the issue): writes the after-action report from job logs: freeze length, before and after load tests, holds and deploys, follow-ups, and a draft of the outage-map downtime notice Texas regulators require (16 TAC §25.52).

Emergency fixes still ship during a freeze through a manual emergency_deploy job, and GitLab records who started it.

How I built it

• GitLab CI/CD: code pipeline (tests, SAST, secret detection, dependency audit, weather watch, build, freeze-aware deploy with a version-checked health check), storm-watch pipeline (hourly schedule on live NWS alerts, or a manual drill), release pipeline on tags. Uses dotenv artifacts, resource groups for safe Cloud Run changes, environments, protected caches and GitLab Releases.
• Weather logic (Python): parses NWS VTEC codes, merges alerts zone by zone, computes freeze windows with per-hazard cooldowns and a 120-hour cap, and fails safe: if NWS cannot be reached it keeps the last red reading and never reports weaker than amber. 105 tests, 99% branch coverage, mypy strict.
• Load testing: Grafana k6 with a realistic traffic mix (page views, area lookups, outage reports) and thresholds on p95 latency, errors and dropped requests.
• Google Cloud: Cloud Run, Artifact Registry with immutable tags, and keyless login from GitLab through OIDC Workload Identity Federation. No stored keys; least-privilege roles.
• GitLab Duo: two custom flows (flow registry v1), each with a minimal tool set, prompts that treat merge-request and issue text as untrusted data, and marker notes so they act only on their own previous decisions.

Challenges I ran into

• Making the "before" test honest. The first load test passed on normal capacity because the demo was too cheap. I gave each lookup realistic work and a realistic traffic mix, so the failure on normal capacity is real, not staged.
• A freeze that cannot be undone by accident. Pipeline variables cannot override it; a drill never scales production down if real weather is amber or red; a drill-tagged commit cannot lift a real freeze; a deploy refuses a storm level older than one hour.
• Agent safety. GitLab rejects flow prompts with HTML comments, so the agents' markers are visible text. Flow triggers fire only for people, so the Steward never closes the issue itself; a person reviews the storm and starts the postmortem.
• Agents that run every hour. The Steward runs after every pipeline, so it needs clear stop rules: it posts only when the level, the freeze or a pass/fail result changes, and drill holds are released only by drills.

Accomplishments that I'm proud of

• The full storm cycle ran end to end on GitLab.com with real data: Beryl replay, failed load test, scale-up, passing load test, Draft hold on a risky merge request, release, all-clear and an accurate after-action report.
• Every number the agents wrote matched the job logs exactly.

What I learned

The hardest part of DevSecOps after the code is written is knowing when to be careful. Weather warnings are a public, free and reliable signal for exactly that, and most teams ignore it.

What's next for Stormproof

• More saved storms (Ida, Harvey, the 2021 Texas freeze) and more service types: 911 text-to-web, shelter finders, school-closure sites.
• Use the GitLab deploy freeze API so other projects in a group share one storm freeze.
• Support NWS river forecasts for slower flood events.

Built With

• artifact-registry
• docker
• fastapi
• gitlab
• gitlab-ci
• gitlab-duo
• google-cloud
• google-cloud-run
• k6
• national-weather-service
• pytest
• python