FairScore - Understanding Loan Fairness Using ML

Front Page
Approval Rating Graph (Equal Opportunity)
Policy Simulation
Chatbot

Inspiration

Taking inspiration from a previous project and a paper by a team member that explored the equal opportunities versus equal outcomes debate in the financial sector, we decided to develop ClearScore—a machine learning tool that simulates and compares loan approval algorithms under both fairness paradigms. Using this machine algorithm, we can work towards determining a refined loan approval algorithm.

What it does

ClearScore uses past HMDA to determine approval ratings for different ethnic groups and analyzes approval trends using an equal opportunity-based approach and an equal outcome-based approach. It then determines the "ClearScore" of each approach, a statistic that we created within the program to quantify the "fairness" of a loan approval algorithm. This single metric provides a transparent way to evaluate and compare algorithms across diverse groups.

How we built it

We built ClearScore using primarily Python and its common libraries including Pandas, NumPy, Matplotlib, and Seaborn. We started with historical HMDA data and obtained a cleaned version of it that included important numbers we needed for our algorithms, such as debt-to-income ratio and income. Then, we created two algorithms: one that only uses these numbers (equal opportunity) and one that also looks at sensitive data (equal outcomes). Then, we developed a statistic - a ClearScore metric that shows how fair each method is by comparing deviations of approval rates across ethnic groups. We also built a dashboard to show our results in an easy-to-understand way. Finally, we added an interactive feature to the program that allows the user to chat with an AI bot to compare graphs from the two analyzed algorithms. Streamlit was used to design the interface of the program.

Challenges we ran into

Given the amount of data visualization that our project has, it was difficult to think of a way to organize the data such that it would be easy for users to understand. Eventually, we were able to figure out an efficient way to organize our graphs and accompany them with tables that represented approval ratings. We also had difficulty deciding on and calculating the right fairness metrics, especially when comparing an equal opportunity approach to an equal outcomes one. Comparing the two algorithms is tough because they have such different intentions (in terms of determining whether to approve or deny a mortgage) when applied to a set of mortgage applicants.

Accomplishments that we're proud of

For two of the members in our group, it was their first hackathon! We all learned a lot about developing an executable program and more specifically using a tool such as Streamlit to develop an interface for it. We were also proud to be able to create a program to perform a novel form of analysis to address a pertinent issue that a few of the members have researched in the past.

What we learned

As mentioned earlier, using Streamlit was completely new to us and being able to build an interface using external software was a really beneficial skill that we learned. Additionally, most of the team members had not previously developed a statistical analysis program going into the hackathon , so it was interesting to develop our own methods for statistical analysis as well as developing a new metric for calculating the fairness of approval ratings (the ClearScore).

What's next for ClearScore - Understanding Loan Fairness Using ML

We plan to expand ClearScore in several ways. First, we would like to add more fairness metrics to create more points of comparison for the algorithms. We would also like to take more financial factors into account when simulating both equal opportunity and equal outcome algorithms, because they can give us a more realistic simulation for each algorithm. This would be especially helpful with a factor such as a credit score, because it is so important in predicting the action taken on a loan.

Demo Video: https://drive.google.com/file/d/1j6S4Cp9jfi8836uI19mGGEx828_LGq3R/view