gender pay gap
biased prediction
before de-biasing
after de-biasing

Inspiration

The gender wage gap is well established -- a wealth of compelling rhetoric demonstrates this effect across a vast number of industries. On average, women make just 78 cents to a white man's dollar, and this effect is even more pronounced for women of color. Equally concerning is the recent literature that demonstrates that men are shown more ads for higher paying jobs than women are which only perpetuates the cycle. This problem is due in part to the use of machine learning algorithms used in the hiring process that are prone to systematic (and very problematic) bias against women.

ProPublica recently conducted a study on a tangential problem involving AI software used to predict the likelihood that a criminal would re-offend. This system was prone to systematic bias against marginalized groups (particularly blacks and Latinos), often significantly overestimating their risk while underestimating the risk that a white man would re-offend.

These examples demonstrate a compelling need to ensure that machine learning algorithms are fair and unbiased. This is a particularly difficult and relevant problem given the increasing prevalence of big data and artificial intelligence systems. In fact, the Effective Altruism community defines AI risk as one of the most important global issues we currently face.

What it does

We created a toolkit aimed at helping developers and companies mitigate bias in their artificial intelligence applications. We analyzed several research papers and selected some of the highest performing de-bias algorithms. Here is an overview of two algorithms we implemented:

XGBoost: This algorithm works by estimating (empirically) the individual impact that each feature variable has on the final prediction. Once we do this, we can cancel out the impact of any protected variables that we don't want the algorithm to use in its prediction (this might include features such as age, race, gender, or physical appearance). Our application of XGBoost on a sample experience versus salary dataset demonstrates strong and consistent performance. We can see that any sign of prejudice is completely removed and that the model faithfully captures the variation driven by years of employment after applying this method.
Adversarial De-biasing: This algorithm works similarly to the adversary in a Generative Adversarial Network (GAN). Essentially, the adversary tries to force the algorithm to make predictions independent of protected variables. The algorithm performs well because it needs to be robust enough to fool the adversary.
Reweighing: This algorithm modifies the weights of different training examples to place stronger training emphasis on less biased ones. This allows the algorithm to mitigate the impact of strongly biased data points on the final model.

How we built it

We implemented our toolkit and algorithms entirely in Python on Jupyter Notebook, using libraries such as numpy, pandas, scikit-learn, and tensorflow.

Results

Our results were very promising! XGBoost gave us near perfect results: controlling for experience, men and women receive around the same predicted salary. From the graphs above, you can see an initial ~$20K wage gap, but after applying the model the prediction lines are almost identical. The reweighing algorithm, implemented through IBM AFI360's toolkit, achieves significant reductions in bias metrics across the board. Average odds difference decreased from a magnitude of 0.1788 to 0.0858, and equal opportunities difference decreased from 0.3576 to 0.0151 (where a value of 0 represents complete fairness). We do see a slight trade-off in the Theil index (increased from 0.0134 to 0.0217) however, which IBM notes (results can sometimes be a mixed bag, but the net impact of de-biasing is positive).

Challenges we ran into

One of the first main challenges we ran into was defining our problem scope. We wanted our toolkit to be applicable to virtually an AI system (since any algorithm is subject to bias), but we also wanted to demonstrate its capabilities on a particular problem. Since we came together at an all-female and non-binary hackathon and many of us are women of color, we decided to pick a salary prediction task (given gender and years of experience) as our example task.

The next major challenge we had was finding a robust dataset. It was difficult to find a dataset that met our needs and contained experience, gender, and salary, so we decided to generate our own data. This was reasonable since the goal of our project (given our timeframe) was to create a proof of concept for our toolkit. We researched salary distributions for men and women in the tech industry and used a multivariate distribution to generate our data.

Another challenge we faced was interfacing with IBM's AIF360 toolkit. The toolkit unfortunately only allowed us to use binary datasets, so we needed to convert our salary label into a binary one. We created our own algorithm for doing so: after fitting models to male and female data separately, we compared the true salary value of each data point with the value our models predicted for the opposite gender. If the difference between the two salaries differed by over 5%, we labeled the data point as "unfair".

Ultimately, though, these challenges allowed us to reconsider our goals and refocus our efforts efficiently. None of us had any experience with IBM's toolkit or de-biasing AI algorithms in general (although we are all very passionate about it!), so this was a great learning experience!

Citation

Angwin, Julia, et al. “Machine Bias.” ProPublica, 9 Mar. 2019, www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
Wiggers, Kyle. “MIT CSAIL Researchers Propose Automated Method for Debiasing AI Algorithms.” VentureBeat, VentureBeat, 25 Jan. 2019, venturebeat.com/2019/01/26/mit-csail-researchers-propose-automated-method-for-debiasing-ai-algorithms/.
“Gender Pay Gap Ratios, Stats and Infographics.” PayScale, 12 Apr. 2018, www.payscale.com/data/gender-pay-gap.
Datta, Amit, et al. “Automated Experiments on Ad Privacy Settings.” Proceedings on Privacy Enhancing Technologies, vol. 2015, no. 1, 2015, pp. 92–112., doi:10.1515/popets-2015-0007.