Inspiration
Fraud detection is a critical part of many businesses today, especially in finance and e-commerce. We wanted to explore how machine learning could help prevent fraud by identifying suspicious transactions, while reducing as much as possible the false alarm cases. The challenge lies, therefore, in catching as much fraud as possible without wrongly flagging legitimate users, a delicate balance that directly impacts trust, cost, and user retention.
What it does
Our project uses a machine learning model to detect potentially fraudulent transactions. It analyzes transaction data alongside user and card information to classify whether a transaction is likely to be fraudulent. The model is optimized for high recall to ensure it catches most fraudulent behavior, while keeping a reasonable balance with precision to avoid too many false alarms. It is important to notice that we also evaluated the model through other criteria, as the recall score alone can result in suboptimal models.
How we built it
The first step to create the model was to extract all the data in the CSV and JSON files into pandas databases. After that, we could merge them all into a single database, grounding the joints by the ids. After that, we had to modify some of the data, being careful in converting NaNs to strings and numerical values into floats and ints. It is worth noticing that the IBM Watson platform dealt with the categorical data processing.
We removed ID columns and full transaction dates. We also excluded the ZIP code, as city and state already provided useful location data. Lastly, we dropped the card number itself, relying instead on more general features like expiration month and year, which proved more informative.
For the classification task, we used the Random Forest and LightGBM algorithms. Our main goal was to optimize for recall, ensuring that the model identifies as many fraudulent cases as possible. We evaluated several model pipelines and selected the one with the highest F1-score, which balances precision and recall. The best model achieved an F1-score of approximately 65%, with decent performance in both recall and precision.
This F1-score becomes especially important in fraud detection, as it reflects the model’s ability to catch fraud without generating too many false alarms.
Challenges we ran into
One of the main challenges was understanding the trade-off between false positives and false negatives. When you increase recall (catch more fraud), you usually sacrifice precision, which can lead to more legitimate customers being wrongly flagged. On the other hand, being too conservative increases the risk of real fraud going undetected.
We also had to be careful about which features to include in the model — keeping only the most useful ones while avoiding data leakage or overfitting.
Accomplishments that we're proud of
We’re proud that we were able to train a model that achieved a solid F1-score while maintaining interpretability. We also learned how to carefully clean and engineer features from raw datasets, a valuable skill when working with real-world data.
Most importantly, we built something that aligns well with practical use cases and shows real potential to be applied in fraud prevention systems.
What we learned
We gained a deeper understanding of how important it is to balance recall and precision, especially in domains like fraud detection where the costs of mistakes can vary widely depending on the business.
For example, if a company sells low-margin products like food online, each fraudulent transaction can be very costly, so they might prefer a model that casts a wider net (higher recall). In contrast, a high-margin SaaS business might care more about not blocking legitimate users, even if it means letting a bit more fraud through.
We also learned how to properly preprocess data, evaluate different model types, and interpret classification metrics in the context of business trade-offs.
On top of that, we found the IBM environment extremely helpful for deploying and testing machine learning models quickly. Its user-friendly interface, integration with popular frameworks, and support for collaborative work made it much easier to iterate and improve our models throughout the project.
What's next for Bigode
Next, we’d like to improve our model further by experimenting with techniques like SMOTE for handling class imbalance, or incorporating user behavioral features like transaction velocity or location consistency.
We also plan to deploy the model into a simple dashboard that shows flagged transactions in real time and allows for manual review. Eventually, we’d like to explore model explainability tools like SHAP to give businesses more insight into why a transaction was flagged.
Log in or sign up for Devpost to join the conversation.