J-LLM

Inspiration

Data analysis seemed fun getting into it and learning more about it here. Learning how to train a model is very fascinating, but Jadco's challenge was pretty difficult. We had to get lots of help, but it turned out ok.

What it does do

It audits and fixes a bursary model that learned to penalize applicants from remote regions, then makes decisions for 4,000 new applicants with a budget that funds 40% of them.

The fairness gap drops from 0.225 to −0.056.
Deserving applicants in remote regions are now funded 94–95% of the time instead of 76–79%.
We also deliver a bias audit, the corrected model, a monitoring plan, a model card and a dashboard.

How we built it

Tools: Python, pandas, statsmodels, scikit-learn and fairlearn.
Measuring the bias: a logistic regression of the old committee's decisions, with one explicit "remote region" term. That term captures the whole penalty as one number, worth about 1.6 R-score points.
The fix: we remove that term when making decisions, so everyone is scored as if they lived in a big city. The top 40% get funded.
Measuring fairness: we use equal opportunity, compared against independent benchmarks. We never use the committee's own decisions as the right answer, since those are what's biased.
Testing: we compared against fairlearn's sosting, with 1,000-resample bootstrap

Challenges we ran into

Deleting the region column doesn't work. Postal code and distance predict the region almost perfectly.
No ground truth. The only labels were the biased committee's decisions, so we had to build independent benchmarks.
Hidden scoring standard. We didn't know how judges would define "deserving", so we tested against six plausible standards.
Standard tools fell short. ExponentiatedGradient left a 0.12–0.20 gap because it copies the biased committee. ThresholdOptimizer closed the gap but lost 5 points of accuracy and used the region to make decisions.

Accomplishments that we're proud of

We turned a hidden bias into one number we can explain, remove and monitor.
The gap went from 0.225 to −0.056, costing only 1.5 points of accuracy (94.3% → 92.8%), with exactly 40% funded.
No remote region is left behind, and the result holds for any budget from 36% to 44%.
The region is never used to rank anyone. Scrambling it changes 0 of the 4,000 decisions.
The simple model fit better than gradient boosting (cross-validated log-loss 0.261 vs 0.280).
Strong results in our own simulation of the judges' scoring: at least 27.3/35, against about 11 for the original model. This is our simulation, not an official score.

What we learned

Spending lots of time debugging, understanding issues, figuring out what the root cause of a problem is, and coming to terms that not all bits of information are relevant.

What's next for J-LLM

We all said we were interested in learning more about AI and ML. We'll keep learning about these topics and, one day, not need so much help from AI itself.