AI Fraud Detection

AI-powered fraud detection with explainable, regulation-grounded compliance reports for Pakistan's financial sector.

Comment

Inspiration

Financial fraud detection in Pakistan's fast-growing branchless banking sector faces a real gap: most fraud systems are either rigid rule-based checks that get bypassed easily, or opaque ML models that flag a transaction as "suspicious" with no explanation and no connection to actual regulation. Compliance analysts are left doing manual cross-referencing against State Bank of Pakistan (SBP) regulatory documents by hand, which is slow and error-prone. Over this hackathon, I wanted to build something that closes that gap and a system that doesn't just say "this is fraud," but explains why, in terms an analyst can act on and cite.

What it does

AI Fraud Detection is an end-to-end console for real-time transaction risk scoring, built solo over two days:

  1. Detection engine: An XGBoost classifier trained on the PaySim synthetic mobile-money dataset (6.3M transactions, 0.13% fraud rate) scores each transaction, achieving 99.95% AUC-ROC and 99.76% recall on held-out test data.
  2. Explainability: Every prediction comes with a SHAP waterfall breakdown showing exactly which features (amount, balance changes, transaction type) pushed the score up or down and no black box.
  3. Regulatory grounding: A hybrid RAG pipeline (BM25 keyword search + dense vector retrieval + cross-encoder reranking) searches real SBP AML/CFT and Branchless Banking regulation documents, then generates a structured compliance report and STR/CTR filing determination, applicable regulations with document/page citations, compliance actions, and recommended next steps.
  4. Conversational compliance assistant: Analysts can ask follow-up questions about a transaction or SBP rules directly, grounded in the same retrieved regulatory context, with a grounding-confidence indicator so answers are never presented as more certain than the evidence supports.
  5. Full transparency tooling: batch CSV scoring, model comparison across 4 classifiers, and an ablation study isolating exactly how much each pipeline component (fraud simulation, SMOTE ratio, feature engineering) contributes to performance.

How I built it

With a tight two-day window, I prioritized getting a working end-to-end pipeline first, then iterating on quality. I started with heavy exploratory work on the PaySim dataset and given a fraud rate of just 0.13%, naive accuracy is meaningless, so I built a two-stage imbalance strategy (a custom fraud-simulation augmentation step followed by SMOTE oversampling) evaluated primarily on Precision-Recall AUC rather than accuracy. Four models (Logistic Regression, Random Forest, Neural Network, XGBoost) were trained and benchmarked with k-fold cross-validation using imblearn's pipeline to avoid data leakage, and I ran a full ablation study to isolate which pipeline decisions actually mattered.

For explainability, SHAP's TreeExplainer generates per-transaction feature attributions. For regulatory grounding, I processed real SBP regulatory PDFs into a chunked, embedded knowledge base in ChromaDB, combined semantic and keyword retrieval, reranked with a cross-encoder, and passed the top results to an LLM (Cohere Command R+) with a tightly constrained system prompt:

if grounding_score < 0.3:
    return "Limited Grounding and Verify Independently"

This required the model to ground every claim in retrieved text and explicitly flag when context is insufficient and no hallucinated regulations.

The frontend is a Streamlit dashboard I rebuilt from scratch with a fully custom dark theme, including a landing page structured around the problem the system solves, an interactive prediction console, and dedicated pages for model performance and ablation analysis.

Challenges I ran into

Class imbalance ordering: getting the fraud-simulation-before-SMOTE sequencing right, and confirming via ablation that this order actually mattered, took real iteration and an easy mistake here silently inflates metrics through leakage. RAG grounding failures: early versions of the retrieval pipeline had a category filter mismatch that silently excluded most of the actual regulatory corpus for certain transaction types, causing the system to falsely report "no evidence found" even when relevant regulations existed. Debugging this required tracing the issue all the way from the UI symptom back to a single mismatched category string. Balancing groundedness with usefulness: an early, overly strict prompt caused the assistant to refuse to answer even when it had good retrieved context, just because a transaction happened to be compliant rather than a clear violation and tuning the system prompt to distinguish "no evidence" from "evidence of compliance" was a genuinely subtle fix. Racing the clock: with only two days, every debugging session had to be fast and decisive and several bugs (a broken try/except structure, an unreachable dead code block, a silent retrieval filter mismatch) required tracing symptoms back to root cause under real time pressure.

What I learned

Building a genuinely grounded RAG system is much harder than wiring up an LLM with retrieved context and most of the real engineering work is in making the system know when to say it doesn't know, and in building retrieval that actually surfaces what's relevant rather than what merely looks relevant. I also came away with a much sharper sense of how class imbalance handling decisions compound through a pipeline, and how easy it is for a metric to look good for the wrong reasons and and how much discipline it takes to debug methodically instead of guessing when the clock is running out.

What's next

Extending the regulatory corpus beyond SBP AML/CFT and Branchless Banking regulations, adding multi-transaction pattern detection (structuring/smurfing across multiple related transactions rather than single-transaction scoring), and building out the batch analysis workflow into a full case-management view for compliance teams.

Built With

Share this project:

Updates

Submission history