US Heat and Cold Plan Audit

Inspiration

Every summer US cities and states publish heat plans. They open cooling centres, send out alerts and run awareness campaigns. Hardly anyone asks whether the measures in these plans are the ones that save lives.

Burke, Wilson et al. (2026) reviewed the causal evidence on temperature and mortality. Their verdict is uncomfortable:

Some of the most popular measures, such as cooling centres, tree planting and cool roofs, have no quasi-experimental evaluation of their effect on deaths.
Some measures plans rarely use have strong evidence: health insurance, income support and affordable heating.
One common measure has evidence of harm. Asking residents to save power during a heatwave raised heat deaths in Japan after Fukushima.

We wanted every resident of California and Texas to be able to ask: my county's plans, as written, against what the evidence says works and against the deaths that heat and cold actually cause here. How big is the gap?

What it does

The app opens on a county map of California and Texas. Each county is coloured by its residents' gap: the extra deaths a year, per 100k people, that the best mix of measures would avert at the same cost as the plans that cover them.

Click a county to open the Strategy Studio:

a Pareto chart of cost against deaths averted, with the plan as written marked ★;
a brief: "+N lives/yr at the same cost: change these, add these, protect these groups";
switches to turn strategies on or off, change their thresholds or change their coverage, with the frontier recomputed live.

Every number links to its source. That is a verbatim quote highlighted on its page in the plan PDF, an evidence-table row, a dataset, or an assumption flagged in amber.

We coded 50 plan documents (state, county and city heat plans, hazard mitigation plans and after-action reports). The app analyses 326 jurisdictions end to end. Each jurisdiction is scored on its plan stack: its own plans plus the plans it inherits from its county and state.

How we built it

The system has three layers, and only the first one is specific to a plan.

1. Codification: a fixed questionnaire per PDF. A blind Claude coder answers the same codebook for every document, using only the codebook, the 37-class action taxonomy and the page text. The codebook has three parts:

the document: its type, year, hazards, lead agency, and whether its thresholds use local health data;
each trigger: its metric, value, duration, and whether it uses a forecast or observed conditions;
each commitment: its action class, when it runs, its target group, who is responsible, how firm the language is, funding and deadline.

Every answer is backed by a verbatim quote and its page. When the plan is silent, the answer is "not stated". The coder never infers.

2. A shared database and taxonomy. All extractions flatten into one SQLite database. The taxonomy links each action class to its evidence grade and effect band from the paper, to a relative cost tier, and to the share of deaths in each target group.

3. A generic engine with no jurisdiction-specific code. A lint test fails on any FIPS code, document id or place name in engine code. Harm comes from population-weighted daily county temperatures (1999–2024, EpiNOAA), EPA CIRA temperature–mortality curves and a CDC WONDER mortality baseline.

An option is a measure paired with an activation rule (for example "daily high ≥ 105°F" or "NWS heat warning"). Its cost and its risk reduction on day $t$ are

$$ x_i = \left(c^{\text{fixed}}i + c^{30}_i \cdot \tfrac{d_i}{30}\right)\kappa_i, \qquad r{i,t} = s_i \,\kappa_i\, e_i \cdot \mathbb{1}[\text{rule}_i(t)] $$

Here:

$d_i$ is the number of days a year the rule is active;
$\kappa_i$ is coverage;
$s_i$ is the target group's share of deaths;
$e_i$ is the evidence-based effect.

A portfolio's deaths averted combine options without double counting:

$$ y = \sum_t D_t \left(1 - \prod_i (1 - r_{i,t})\right) $$

We answer three questions on the cost–deaths frontier:

Retime: what if the plan kept its own measures and only changed when each one switches on?
Reallocate: what if the same budget went to any mix of the menu's measures?
Add standing policies: what do year-round policies, such as income support and insurance, add?

Inherited plans form a fixed floor, and each jurisdiction's own measures are scored on the deaths that remain.

The headline counts only measures with causal evidence. An "if unproven measures work" view shows the upside.

Stack: Python (pandas, NumPy, SciPy), FastAPI, Plotly and pdf.js. The app has no build step.

What we learned

Cold matters as much as heat, and plans mostly ignore it. Many of the largest gains in California come from cold measures. The state's plans have no cold trigger we could convert into an activation rule.
Timing is a cheap win. In the evidence-only view, retiming California's state plans takes them from about 30 to about 51 deaths averted a year. Most of that gain comes from confining harmful conservation appeals to the most extreme days.
The biggest per-resident gaps are in small rural counties that have no plan of their own, such as Siskiyou, Lake and Inyo. They inherit only the state's plans.
An LLM can code policy documents reliably if it only answers questions. A blind second coder agreed on action class with $\kappa = 0.78$ and on commitment strength with $\kappa = 0.84$. Status agreement was low ($\kappa = 0.38$): "planned" and "proposed" are hard to tell apart.

Challenges

Making the result independent of how a coder splits bullets. Different coding passes split the same plan's bullets very differently. One pass coded a single track as 120 commitments. Another coded the plan's progress summary as mentions only. We made the unit of analysis the shared-taxonomy measure (action class × target group × hazard), not the bullet.
Trusting the coder. We built quote verification (100% of quotes found on their cited page), recall checks against the plan's own action labels (126/126 for the California plan), and injected-error tests. We also built an eval maker: reviewers see the PDF beside each AI answer and mark it correct, fix it or reject it.
Converting triggers into rules. A plan's "excessive heat warning" has to become a set of days in each county. California's NWS zone mapping is unreliable before 2022, so we fall back to ranking each county's hottest days. Some triggers ("an extreme heat event") cannot be converted, and we flag them.
A search space of about $10^{22}$ timings. We use multi-start greedy search with a Pareto filter, checked against a random sample of the whole space.
Keeping the LLM out of the arithmetic. Joins, burden and frontiers are deterministic code. The model only reads PDFs.
Honest limits.
- Effect bands and cost tiers are team assumptions. Costs are relative tiers, not dollars.
- The coding has not yet been checked by humans.
- Texas mortality cells with 1–9 deaths are suppressed in CDC WONDER and count as 0.