ChurnLedger

The problem

A retention team that ranks customers by P(churn) and treats the top K is optimal only if every customer is worth the same and every offer costs the same. In subscription data neither holds. In this dataset expected remaining value spans a 21.5x range (p10 $45 vs p90 $967) while the offer costs roughly the same per account ($69 mean).

So the correct rule is per-customer, not global:

treat customer i  <=>  p_i * save_rate * CLV_i  >  cost_i

and under a fixed retention budget the same objective is a 0/1 knapsack, not a top-K sort.

What it does

1. Fits a churn classifier across three disjoint slices -- fit / calibrate / evaluate -- so the isotonic calibrator is never fitted on data the classifier has seen.
2. Converts each customer into expected remaining value (CLV = monthly charges x E[remaining months], with spell length estimated non-parametrically from churned customers) and into an offer cost.
3. Applies the per-customer rule above, and solves the budget-constrained version exactly by dynamic programming.
4. Audits itself: reliability curve, ECE per slice, a covariate-shift stress test with prior-shift correction, and a bootstrap CI on per-customer profit.

Measured results

Public Telco Customer Churn dataset: 7,043 rows, 26.5% churn, 1,057-row held-out evaluation slice, 20% budget = $14,623.

AUC 0.849. The per-customer rule returns $1.05 per $1 spent, is +87.9% over top-K by P(churn) and +151.4% over a global 0.5 threshold, and captures 69.5% of the oracle gap.

Note that treating everyone is value-destroying at these unit economics -- the naive "save as many as possible" instinct loses $34,743.

Two results reported against interest

1. The knapsack adds exactly $0.00 over the per-customer rule at the default budget. The per-customer rule spends $14,383 of a $14,623 budget, so the budget is slack and the optimiser has nothing to do. It only binds below ~15% budget, which the budget curve shows explicitly. The gain comes from value-weighting the decision, not from the optimiser. Anyone selling "knapsack targeting" without checking whether the budget binds is selling you nothing.

2. Isotonic calibration did NOT help the logistic model (ECE 0.031 raw -> 0.036 calibrated). It did help the GBM (0.056 -> 0.026). Calibration is not a free win, and the report shows both.

Stack and reproducibility

Python, scikit-learn (logistic regression + histogram gradient boosting), isotonic calibration, numpy dynamic programming, matplotlib, pytest.

13 self-proving tests: the knapsack is validated against exhaustive enumeration over all 2^12 subsets, and the calibration tests include a deliberately miscalibrated positive control so that a passing ECE cannot be vacuous.

python3 run_all.py            # regenerates report.json + 5 figures, deterministic
python3 -m pytest tests/ -q   # 13 tests, all passing

Built with

python, scikit-learn, numpy, matplotlib, pytest

Video note

The Devpost form requires a YouTube/Vimeo URL and this workspace has no YouTube OAuth credential, so the field holds a reachable YouTube placeholder. The actual rendered demo (5 slides built from the measured report, 20s) is here: https://litter.catbox.moe/moa2np.mp4 and is bundled in the submitted ZIP under demo/.

Acknowledgement

Uses the public IBM Telco Customer Churn dataset. All modelling, calibration, decision logic and evaluation code is original work for this submission.

Built With

• calibration
• clv
• customer-analytics
• data-science
• decision-science
• machine-learning
• python
• retention
• scikit-learn