MULTIMODAL SPAM CLASSIFICATION USING TEXT AND IMAGE ANALYSIS

About the Project

Spam and scam messages are becoming increasingly sophisticated, appearing not only as text messages but also through images and suspicious URLs. This inspired us to build Multimodal Spam Classification Using Text and Image Analysis, an AI-powered cybersecurity solution designed to analyze different forms of potentially harmful content in one system.

💡 Inspiration

Traditional spam filters often focus primarily on text. However, modern scams can hide malicious information inside images, QR codes, or URLs. We wanted to explore how Artificial Intelligence and Machine Learning could be used to analyze multiple types of content and provide users with a clearer understanding of potential threats.

🛠️ How We Built It

The project currently focuses on a machine-learning-based text classification pipeline using the UCI SMS Spam Collection dataset. We implemented text preprocessing, TF-IDF feature extraction, and Logistic Regression for spam classification.

The system also includes a URL analysis module that extracts URLs from messages and evaluates them using multiple risk indicators, including suspicious domains, IP-based URLs, URL shorteners, excessive subdomains, suspicious keywords, unusual ports, and other potentially unsafe characteristics.

We also incorporated confidence scores and explainability features to make predictions easier to understand rather than simply returning a spam/not-spam label.

📚 What We Learned

Through this project, we gained practical experience in:

Natural Language Processing (NLP)
TF-IDF feature engineering
Machine Learning classification
Model evaluation and performance analysis
URL security and risk analysis
Explainable AI
Python project architecture
Dataset preprocessing and model persistence
Building an AI-powered cybersecurity workflow

Our text classification model achieved approximately 97.13% accuracy on the test dataset, demonstrating the effectiveness of the implemented baseline approach.

🚧 Challenges

One of the biggest challenges was designing a system capable of handling different input modalities while maintaining reliable predictions. The available spam dataset primarily contained text messages and did not provide scam-category labels or multimodal image data. This required us to carefully separate the implemented components from future extensions.

Another challenge was making the model interpretable. Instead of treating the prediction as a black box, we explored ways to identify the features and indicators contributing to suspicious-content detection.

🚀 Future Scope

The project can be extended with OCR-based image analysis, QR-code detection, image classification, multimodal model fusion, scam-category classification, multilingual support including Tamil, and an interactive dashboard for prediction history and analytics.

Ultimately, our goal is to develop a more comprehensive and explainable AI-based cybersecurity assistant that helps users identify suspicious digital content before it becomes a threat.