Two Cakes

Inspiration

Most carriers treat retention as a trade-off. You can hand out discounts and lose money, or you can save money and lose customers. We didn't think you had to pick.

The two cakes are the customer and the business. Customers shouldn't get spammed with offers they don't need, and T-Mobile shouldn't pay to keep people who were staying anyway. Research backs this up: in one large wireless field study, proactive plan outreach actually raised churn, and the riskiest customers and the most persuadable ones only overlap about half the time.

A churn score tells you who might leave. It doesn't tell you what to do, and a lot of the time the right answer is to wait. So we set out to build the decision, not just the score.

What it does

Two Cakes scores every customer's chance of leaving in the next 90 days, shows the reasons behind it in plain English, and then picks one action for that customer, where doing nothing is allowed to win.

Who's drifting: a risk score for every account, with the top three reasons to leave and the top three reasons to stay.
The deal engine: every move a customer qualifies for is weighed against doing nothing, which is always on the list at $0. A move has to beat it to be sent.
Fix first: if a customer has an open problem, like an unresolved coverage issue, the only message allowed is a fix. No sales pitch.
No upsells. We never try to sell more to someone who's about to leave. A cheaper plan with the same features counts as a save, and it goes on the bill as a tap-to-confirm line rather than in an unprompted text.
Their channel, their language: the exact message, in English or Spanish, by text, in-app note, email or call, whichever the customer prefers. It shows on a phone mockup, and a button sends it to a real phone as a push notification.

The numbers: 5,042 customers scored, 505 flagged, 184 messaged, and 4,858 left alone on purpose. The riskiest 10% of customers hold about 62% of expected leavers.

The live demo: follow customer #02363 as events come in, and watch the decision change:

Event	Churn risk (90 days)	Decision
Start	5.83%	Wait
1. Logged into the app	5.62%	Wait
2. Dropped call, with a coverage chat still open	9.07%	Wait
3. Carrier-shopping score up	18.42%	Wait
4. Asked about moving their number	26.31%	In-app fix-first message, English or Spanish

For the first three steps the engine holds back, because the best offer would cost more than it's worth. At event 4, a fix-first note finally beats doing nothing, so it goes out in the app with no sales pitch. The page shows which reason moved the score at each step and why the engine is still waiting.

How we built it

We built on the judges' T-Mobile sandbox: 5,042 accounts with bills, usage, support history, risk signals, and real plan and promotion rules.

Labels. The real churn labels are private, so we made our own stand-ins. We extended the judges' public example formula from 7 terms to 21 (unresolved vs. negative support contacts, early tenure, app use, promo expiry and more) and set the overall level to T-Mobile's reported postpaid churn of about 1% a month.

The model. A plain logistic regression on 20 fields. It explains itself exactly: each field's push on the score turns into the top reasons to leave and to stay.

How good is it, honestly? Scored against the judges' public example model's churn probabilities, our out-of-fold AUC is 0.797. The best possible score there is about 0.825.

What we tried. We didn't stop at one model. Every candidate was scored the same way:

Model	AUC vs judges' example model
Logistic regression (shipped)	0.797
Spline logistic regression	0.806
Survival model	0.802
Explainable boosting	0.794
Gradient boosting, tuned	0.786–0.789
Simple rule: rank by support calls	0.776
Best possible	about 0.825

Fold-to-fold noise is about ±0.03, so nothing fancier beat the plain model by more than noise. We kept the simplest one: it's well calibrated, which matters because the deal engine multiplies its probabilities by dollars, and its reasons are exact.

The deal engine. For each customer it lists 15 possible moves. Real eligibility rules and our guardrails rule out most of them before any math. Each remaining move is priced as chance of leaving × chance the move keeps them × what keeping them is worth, minus the offer's cost, agent time and a small charge for bothering them. If nothing beats $0, we wait. Messages come from templates, and every dollar amount in them is checked against the real plans and promotions tables.

The app. Python, pandas, scikit-learn and Streamlit, as one scroll page. No LLM in the product, so every decision is predictable, cheap and rule-checked.

Challenges we ran into

Accuracy is a trap. With about 1% monthly churn, predicting "nobody leaves" is 99% accurate and useless. We measure how well we rank risk instead.
No real outcomes. The real labels are private, so we built our own and were careful about which scores mean something. Our model also scores 0.85 against our own labels, but those labels come from the same fields, so we report 0.797 against the judges' model as the honest number.
The save rate is an assumption. Nobody knows yet how many messaged customers actually stay because of the message. We use 10% as our central assumption and show where it breaks even, so the business case doesn't hide behind one number.
Messages that quote prices have to be right every time. Every amount is checked against the plan and promotion tables before it can be sent.

Accomplishments that we're proud of

Restraint you can check: 4,858 of 5,042 customers get no message at all.
The deal engine pays its way. About +$3,900 net at a 10% save rate, and it breaks even at a 5.17% save rate.
A demo you can watch think. Customer #02363 goes from wait to act as their risk climbs, and the page explains every step.
Honest scoring. We scored every model we tried against the judges' example model, and kept the simplest. It lands at 0.797 out of a possible ~0.825.

What we learned

Contact can backfire. High risk isn't the same as persuadable.
A churn score is only an input. The useful part is choosing the action and explaining it.
Show everything you tried, not just the winner. The simple model held up against every fancier one.
Our labels, not our model, are what limit the score. Better labels are the way up.

What's next for Two Cakes

A real holdout test to measure true save rates instead of assuming them.
A feedback loop that learns from what actually happened after each message.
Live T-Mobile signals feeding the live-events view in real time.
A weekly fairness check, so customers at the same risk get treated the same way regardless of age or student status.