Inspiration
Anomaly Detection in Financial Data with Explainable AI
Team Members: Padma Prabagaran, Nilay Laddha, Subham Kumar Das
Introduction:
Anomalies in financial data can be indicators of potential risks or opportunities and being able to detect these can be beneficial for investors, financial institutions, and regulators. This project aims to find any anomalies within the financial markets and uses Explainable AI to help us better understand what might have caused the specific anomalies. We also want to incorporate sentiment from social media to help identify these anomalies. Although this project is geared towards financial data the motivation of this project stems from the idea that a similar model can also be used to detect anomalies in other time series data such as healthcare and patient monitoring, energy consumption, or any signal data. This is an Unsupervised Learning problem since we do not have any labeled data, but rather we will train the model with stock market data and cross-check the anomalies with real-world dates.
Related Work:
There are a couple of papers that have published ways to detect anomalies in time series data and others that have used explainable AI methods:
Papers:
Paper summary: The idea proposed in this paper is to use an LSTM-based autoencoder that encodes the data into lower dimensions and then decodes it to the original space to learn the general patterns of time series signal data. When the model encounters any anomaly in the data, the reconstruction error would be unusually high, which can be used to detect the anomaly. They concluded that to achieve optimal performance in anomaly detection using neural networks, the architecture and post-processing parameters must be tailored to the specific input data. Key parameters include the number of layers and LSTM cells, window sizes for input data and smoothing, and a threshold for detecting anomalies. While other methods may work for specific time series types, the autoencoder approach is versatile and effective for all types of time series.
Blogs: https://medium.com/swlh/time-series-anomaly-detection-with-lstm-autoencoders-7bac1305e713
Data:
We are using publicly available datasets on stock market data and tweet data from Kaggle (link below). It contains the daily stock market prices of the various companies in different indexes. For this project, we will solely be using stocks of companies in the S&P 500. We have stock price data for each company between the year the company IPOed till 2021. The second dataset consists of tweets about top companies between the years 2015 and 2020. We are using the tweets to capture the relationship between the sentiment of tweets and how the stock prices fluctuate. We will need a significant amount of preprocessing to link the stock price data with tweet data by aligning the timeframe from both these datasets.
https://www.kaggle.com/datasets/paultimothymooney/stock-market-data (1.3 GB)
https://www.kaggle.com/datasets/omermetinn/tweets-about-the-top-companies-from-2015-to-2020?select=Tweet.csv (772.34 MB)
Methodology:
We will use a combination of LSTM and autoencoder models for our project. We will initially use Google Colab to test our script on a subset of the data, and then probably use Oscar or CS cluster for our final training and testing. Our design is to predict anomalies by using past stock price trends and Explainable AI techniques to determine if there was a time frame before the anomaly that might be causing the event.
Autoencoder Model: Design an LSTM autoencoder to capture temporal patterns in the stock price data and learn a compressed representation of the input data. The autoencoder will be trained to reconstruct the input data with minimal loss, effectively capturing the underlying structure.
Explainable AI: Use attention mechanisms (or other methods such as LIME/SHAP) in the LSTM model, visualize the attention weights, and understand which parts of the input sequence are most relevant for anomaly detection. We are still figuring out what technique to use.
If time permits use tweets to see if there is a correlation between the detected anomalies and public sentiment.
Metrics:
Since the task is unsupervised, there is no sense of accuracy in our project. We believe that the project can be viewed as successful if the detected anomalies in the stock market data can be made sense of by looking at the points and actual dates of big fluctuations. For the baseline, to get a sense of how well our model performs we will compare our reconstruction loss with that of the one obtained from the paper. Also, as a sanity check, we will compare our model results with pre-existing statistical method results.
Base = Get comparable results to the paper that is our baseline.
Target = train a model to catch anomalies in stock price data, with explainable characteristics associated with that anomaly using attention.
Stretch = predicts anomalies just like in the case of the target model, but along with processing the tweet data alongside it, to come to a good prediction.
Ethics:
Why is Deep Learning a good approach to this problem?
Deep learning is a good approach for our project as financial data often contains intricate patterns and relationships that may need to be more easily discernible using traditional statistical methods. Deep learning models can automatically learn complex representations from the data, capturing both linear and non-linear relationships.
What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain?
All our data is publically available for anyone to see and use and thus there are no concerns about their collection. Since we are using tweets that are about the companies in the S&P 500 there are no obvious historical or societal biases that stand out. But, we can have a bias in the sentiment analysis or media bias that can affect the tone and content of the tweets (if we get to the stretch goal).
Division of labor:
For the most part, we plan to work together for all parts of the project. However, we may adjust the load on days for the flexibility of team members' schedules.
Reflection:
Challenges: What has been the hardest part of the project you’ve encountered so far?
The paper we used as inspiration for this project, mentioned that the number of layers, units, and window size for the LSTM autoencoder is an empirical process and thus we tried various combinations to find the best architecture for our data.
For the interpretation part, we tried implementing SoftAttention from scratch and ran into a lot of problems. Whenever our version of Attention was added to the model architecture, the data reconstruction was flat-lined. We spent a lot of time trying to figure out why this was happening. After a lot of debugging and trying to figure out what was happening, we ultimately decided to use the TensorFlow attention layer. We were able to get better results. We also found that adding LSTM layers over 2 layers also made the reconstruction not work. We are now able to detect some anomalies in the data. A lot of time was also spent trying to figure out how to plot our values and how we should sequence the data for input into the model.
The main challenge right now is to figure out how to visualize the output of our attention mechanism in a way that makes it interpretable.
Insights: Are there any concrete results you can show at this point? How is your model performing compared with expectations?
Yes, we are able to get anomaly predictions from the reconstruction of the test data, but the The interpretation part is where we need to work on. We found out that some of the methods we learned in class and wanted to see if we could try them out for our project are not compatible with our data.
Plan: Are you on track with your project? What do you need to dedicate more time to? What are you thinking of changing, if anything?
We currently have a working model but still need to work on interpreting it and understanding how we can best visualize the results.
We need to work on the interpretability portion of our project and figure out a way to implement an interpretable visualization for our model prediction. We also need to implement a statistical method for anomaly detection.
Check-Off #2: https://docs.google.com/document/d/1YP3uWa86J8O0UhXtspAx-AMTxakX637vsuiBVHLyMmc/edit?usp=sharing
Check-Off #3: https://docs.google.com/document/d/1_5Lrex70Yxz3pQ-vZ3nQOTcVWNI_n2DFntXZUKmkKA8/edit?usp=sharing
Final Write-Up: https://docs.google.com/document/d/1v6bMncUaWReRgKgT4Ll6xpO7D4CDQ-qF2Cm5RxFlOr0/edit?usp=sharing
Slides: https://drive.google.com/file/d/1uHWiSTHmh-15E_BYcpYTXTxjoUhxXkPD/view?usp=sharing
Github: https://github.com/padmapraba/Anomaly-Detection-For-Financial-Markets
Built With
- keras
- python
- tensorflow
Log in or sign up for Devpost to join the conversation.