cometx(multimodal fake news detection system)

COMET-X: The Multimodal Misinformation Detector.

Comment

COMETX - MULTIMODAL FAKE NEWS DETECTION

the main inspiration for this project is derived from the ambition to develop and improve our skills in the path of artificial intelligence and software developement and test out how real-world implementations of software systems

The project is a multimodal analysis application designed to detect misinformation by examining both text and images within a post. The core of the system is a pipeline that processes information in several key stages:

Feature Extraction: The application uses robust pre-trained models to convert raw text and images into numerical representations called embeddings. Specifically, it uses RoBERTa for text and a Vision Transformer for images.

Multimodal Fusion: The individual text and image embeddings are combined into a single, unified feature vector. This fused representation captures the combined meaning and context of the post.

Graph Analysis: A simulated Graph Neural Network (GNN) processes these fused features within a mock social network structure of users and posts. This step simulates how information spreads and is influenced by network dynamics.

External Fact-Checking: The application leverages an external API to cross-reference the text claim against a database of verified facts from trusted sources.

Decision-Making & Explanation: The system's final verdict is determined by prioritizing the external fact-check results. If a definitive "False" or "Misleading" rating is found, it overrides the GNN's prediction, resulting in a verdict of "Fake" with high confidence. The final explanation is a detailed, human-readable summary that combines both the external fact-check verdict and the internal model's analysis.

Challenges we ran into

The main challenges faced are

  1. There are many similar implementations of the same concept of fake news detection, and creating a novel project idea was a first challenge we have faced

  2. it was also faced challenges with fully implementing the gnn part of the fake news detection as it was made in considerations with the time and resources that were avaiable to us

  3. the user interface enchancement challenges was also faced due to allocation of time was maximum towards the overall cometx development

Accomplishments that we're proud of

  1. the Human Explainable Multimodal Misinformation Recognition module is able to identify phishing tactics , scam tatics and is able to work along with the fact checking apis to be more precise

  2. there is also a foundational and finetuned variation of VIT model used to detect if the image is ai generated which can also contribute towards the overall classification of the news provided to us

What we learned

  1. this project has broadened our horizon on working on complex networks and deep learning and making good use of time and resources that we have currently and we have benefitted greatly from it

  2. we were able to learn how actually a misinformation spreads in a network and it has piqued our interest towards how real world applications can be developed as long as we are ready to commit time and effort towards it and working efficiently

What's next for cometx(multimodal fake news detection system)

  1. the main future implementation that we were hoping to intergrate was bot detections system in our cometx itself , as once we know if its a bot or not we can more accurately predict the information and how quick and scripted bots try to perform like humans

2.Expand Multilingual Support was the other idea we want to see be implemented in the furture, The current text-based analysis is primarily for English. By integrating multilingual embedding models and fact-checking APIs, the application's reach could be significantly expanded to analyze posts in different languages.

Built With

Share this project:

Updates