CheckMailAI: Phishing Email Detection

Full website
Main page of Website
Enter Email 1 Website
Phishing result Website
Enter Email 2 Website
Legitimate result Website
Main Software
Results on Software
Enter Email on Software

Inspiration

Phishing emails are responsible for over 90% of successful cyber-attacks worldwide. We were inspired by the lack of transparent, explainable, and publicly accessible phishing detection tools. Most existing systems are either black-box models with no explanation, or single-modality systems that analyse only text or only URLs — never both. We wanted to build something that actually works in the real world and tells users why an email is suspicious.

What it does

CheckMailAI is a real-time phishing email detection system that analyses both email body text and embedded URLs simultaneously. It uses a dual-layer weighted ensemble:

Layer 1 — Enhanced DistilBERT: Analyses email body and subject for deceptive language patterns (99.61% F1, 0.24% False Positive Rate)
Layer 2 — CNN-char + Bi-LSTM: Analyses each extracted URL at character level for structural phishing patterns (97.41% F1 on 2.3M URLs)
Ensemble: Weighted Soft-Voting (60% text + 40% URL) produces the final verdict
LIME XAI: Explains why an email is phishing — highlights suspicious words and URL segments in human-readable language

Deployed as a live web application at checkmailai.app and as a cross-platform desktop software.

How we built it

We trained Enhanced DistilBERT on a master dataset of 135,029 emails from five public corpora (CEAS_08, Nazario, Enron, Nigerian_Fraud, SpamAssassin) using Google Colab with L4 GPU. The CNN-char + Bi-LSTM URL model was trained on 2,305,684 balanced URLs from four sources including PhiUSIIL and Razvan27.

The backend is a Flask REST API deployed on a Digital Ocean Droplet (Ubuntu 22.04, 8GB RAM, 4 vCPUs) with Nginx handling SSL termination. The frontend is pure HTML/CSS/JavaScript. The desktop app uses pywebview and is packaged as a standalone .exe via PyInstaller.

Challenges we ran into

Balancing the dataset across five different email corpora with inconsistent schemas
Preventing DistilBERT from memorising domain strings instead of language patterns (solved via URL masking with [URL] token)
Running LIME on a CPU-only server within acceptable response time (solved by reducing perturbation samples to 30)
Resolving 207 conflicting URL labels across merged datasets

Accomplishments that we're proud of

99.61% F1 Score with only 0.24% False Positive Rate on DistilBERT
97.41% F1 Score on 345,853 unseen URLs for CNN-BiLSTM
AUC-ROC of 0.9993 — near-perfect discrimination
Fully deployed live system accessible to real users at checkmailai.app
Transparent XAI explanations that make AI decisions understandable

What we learned

How to fine-tune Transformer models for domain-specific classification
The importance of URL masking as a pre-processing step for NLP models
How LIME works internally and how to optimise it for production CPU environments
End-to-end ML deployment: from training on Colab to serving via Flask on cloud