MMIFD - Multi-Modal Insurance Fraud Detection

FrontEnd UI
Results Analysis Part 2
Results Brief Part 1

Inspiration

For this project we were really excited to work on integrating multiple SOTA machine learning models to predict property insurance fraud.

What it does

Our application uses a web framework where a user uploads a claim form as well as photos and videos of the property for which property damage is being claimed. These documents are then fed to three task-specific machine learning models which focus on analyzing metadata from photos and videos, analyzing the actual damage presented in the photos and videos, and finally analyzing satellite imagery of the property in question. The outputs from these models are then fed to a general purpose LLM which summarizes findings and provides an overall recommendation as to whether or not the claimant is likely to have committed fraud at some level of their claim.

How we built it

We have four team members, three of whom created the task-specific models while the fourth team member created the systems user-interface. We then collaborated on separate pieces for extracting text data from claimant forms and synthesizing overall results with a general purpose LLM.

Challenges we ran into

The first and most difficult challenge we encountered was dealing with ambiguity around the type of data needed to effectively train models to predict fraud in a way that would be valuable for Assurant's business. Part of this was due to vague requirements - while we appreciated that the challenge contained more details than most of the other challenges presented at Hacklytics 2025, there was really no specific direction around what type of data the Assurant team would like to ingest in the system. Obviously for privacy reasons we could not expect to receive labeled datasets from Assurant containing examples of fraud from among their customer base, however were we to product ionize this application for professional use, we would need to spend significantly more time thinking through detailed business requirements which could then be implemented with our tech.

Accomplishments that we're proud of

We were able to create a truly multi-modal application for detecting fraud in less than 36 hours. While ideally we would spend considerably more time refining business requirements and perfecting user experience, this challenge afforded us an amazing opportunity to work on integrating various SOTA machine learning models which we would otherwise not have good reason to work with (for example: making use of an ML model for analyzing satellite imagery developed by the US DoD's Defense Innovation Unit).

What we learned

We learned a lot about the individual models we chose to use for our system and best practices for integrating said models into one cohesive outputs via LLMs. To note, an LLM may or may not be the ideal method for fusing these sorts of model outputs but given the time constraints we were working under, we found its use to be more than adequate.

What's next for MMIFD - Multi-Modal Insurance Fraud Detection

We would be happy to have a longer discussion with the team from Assurant about how we could further refine their business requirements and produce a system that could potentially help save them a lot of money in detecting insurance fraud.