Inspiration
I came into this project from a research background in LLM demographic bias evaluation in healthcare settings — specifically studying how language models produce different clinical recommendations for identical patients when protected attributes change. The accuracy gaps we found (up to 9% across demographic groups) were alarming enough in healthcare. But when I looked at financial services, the stakes were even higher: fair lending isn't just an ethical concern, it's a federal legal requirement under the Equal Credit Opportunity Act.
The question that wouldn't leave me alone: every major bank now uses AI for underwriting. Every one of those models was trained on historical data that contains decades of discriminatory lending patterns. Who is auditing whether these models reproduce that discrimination — and when are they doing it? The answer, in most cases, is compliance teams running statistical reports months after decisions are made.
I wanted to build something that moved the audit from after-the-fact to before-the-decision. FairLane was the answer.
What it does
FairLane is a fair-lending agent that audits its own credit decisions for demographic bias before a human loan officer signs off.
For every application, FairLane:
- Drafts an underwriting decision using only legitimate financial factors
- Generates counterfactual variants of the applicant — identical financials, swapped race, sex, and ethnicity
- Re-runs itself on every variant and measures whether decisions change
- Scores reasoning consistency using an LLM-as-judge evaluator
- Escalates to a human with a complete audit packet if any variant flips or reasoning diverges across demographic groups
- Monitors its own consistency over time and self-corrects when bias drift is detected
The loan officer sees the full evidence: which variant flipped, the approval rate gap, the judge's consistency score, and a plain-language explanation — then makes the final decision. Nothing is finalized automatically.
How we built it
The agent pipeline runs five sequential steps orchestrated by Google ADK with Gemini as the reasoning engine:
Underwriting reasoner — Gemini drafts a decision from legitimate financial features only. The protected attributes are passed in the payload but the system prompt explicitly forbids using them, and the audit tests whether the model obeyed.
Counterfactual generator — Pure deterministic Python. Takes the application and sweeps one protected attribute at a time, holding all financial features byte-identical. Sweeping one attribute at a time (not the full cross-product) keeps attribution clean and API costs manageable.
Disparity analyzer — All metrics computed in Python, never by Gemini: flip rate, approval rate gap, terms gap. Gemini only runs the LLM-as-judge consistency scoring — evaluating whether reasoning was consistent across variants even when the binary decision matched.
Decision router — Gemini composes the human-facing audit packet. The recommended action (escalate vs safe) is computed deterministically and handed to Gemini to explain — it never recomputes it.
Compliance logger — Full audit record persisted to SQLite: baseline decision, all variants, metrics, traces, and the human officer's final action.
Arize Phoenix traces the entire pipeline as a single unified span tree and powers the drift detection loop through its MCP server — the agent queries its own operational data to detect consistency regression and propose prompt fixes.
Data: public HMDA mortgage records from the FFIEC Data Browser API — real demographic and financial features, no synthetic data.
Challenges we ran into
The "same decision, different reason" problem. Early versions used only flip rate as the bias signal. But the LLM-as-judge kept returning low scores on applications where the binary decision (approve/deny) matched across all variants. Investigating revealed the model was approving the baseline for "strong income" but the demographic variant for "low-risk geography" — identical outcomes, demographically influenced reasoning. Catching this required the LLM-judge scoring rationale consistency, not just decision consistency. That distinction ended up being the most important technical insight in the project.
Positive instructions are not enough. Telling the model "use only these legitimate factors" was insufficient. Without explicit "blindness" constraints — "do NOT use, infer, or proxy any demographic attribute, including geography, name, or zip code" — the model allowed demographic noise to leak into its reasoning. The drift experiments proved this: removing the hard constraints dropped consistency scores from 0.95 to 0.68. Restoring and strengthening them brought scores back to 1.0.
Public data limitations. HMDA's public LAR excludes credit score and the automated underwriting result for privacy protection. This means "holding creditworthiness constant" is approximate — held constant only on the features available. This is stated explicitly in the audit packet and README; overclaiming would undermine the project's credibility with anyone who knows fair lending.
Honest scope. The counterfactual method tests for disparate treatment — the model reacting directly to a protected attribute. It does not, on its own, prove the absence of disparate impact (facially neutral features that correlate with protected class). Building the system to be honest about this distinction was harder than it sounds.
Accomplishments that we're proud of
The drift loop actually worked. Running three prompt experiments and watching the Phoenix evaluation scores move — 0.95 baseline, crash to 0.68 after weakening constraints, recover to 1.0 after the fix — with the agent detecting its own regression through the Phoenix MCP server and proposing the specific prompt change needed, was the moment the system proved itself as genuinely agentic, not just a pipeline.
Application 126. A real HMDA mortgage application: $255,000 cash-out refinance, denied on DTI and lien status. When sex is changed to Female — same income, same DTI, same loan amount — the model flips to approve. FairLane caught it automatically. That moment, live on a real public dataset, is what the project was built for.
The LLM-as-judge insight. Discovering that rationale consistency matters as much as decision consistency — and building an evaluation system that captures both — is a contribution I think has value beyond this specific use case.
Staying honest about scope. In a hackathon context there's pressure to overclaim. Being explicit about what the system does (disparate treatment detection) and doesn't do (disparate impact analysis, certified compliance) while still building something compelling and technically deep felt like the right call.
What we learned
Fair lending is a harder problem than it looks — not just technically but philosophically. Fairness is not a property of data or of models in isolation. It is a property of instructions. The same model, on the same data, with weakened constraints produced measurably worse bias outcomes. The fix was not retraining — it was a single sentence in a system prompt.
That's both encouraging (bias can be mitigated through careful prompting) and sobering (any system without active monitoring will drift). The combination of explicit negative constraints and continuous observability is what makes the system trustworthy rather than just capable.
On the technical side: separating Gemini's role (reasoning, explanation, judgment) from Python's role (metrics, thresholds, deterministic logic) was the most important architectural decision. Every time we were tempted to ask Gemini to compute a number, the result was less reliable than doing it in pure Python. The LLM is a reasoning engine, not a calculator.
What's next for FairLane
Disparate impact layer. The current system detects disparate treatment. The next layer is flagging proxy features — geography, census tract, property value in specific zip codes — that correlate with protected class even when not used directly. This is the harder and more legally significant problem.
Credit score integration. Public HMDA data lacks credit score. Partnering with a lender or credit bureau would allow true financial parity testing — holding creditworthiness fully constant, not just approximately.
Domain generalization. The counterfactual audit pattern is domain-agnostic. Hiring decisions, insurance underwriting, clinical triage, university admissions — any high-stakes AI decision with protected class considerations is a candidate for this architecture. The next build is a hiring agent version.
Regulatory integration. ECOA requires adverse action notices. FairLane's audit trail contains everything needed to generate these automatically — the specific factors that drove the denial, the evidence that they were applied consistently. Automating adverse action notices from the audit trail would make the compliance value immediate and quantifiable.
Multi-model ensemble. Running the same application through multiple LLMs and flagging when they disagree on demographic variants would catch model-specific biases that a single-model audit misses. If GPT-4o approves and Gemini denies when only sex changes, that disagreement is itself a signal worth surfacing.
Log in or sign up for Devpost to join the conversation.