Deep Confusion

Our project would be based on the two papers attached below. Both share the same goal of using daily news sentiment as a feature in an LSTM model to predict stock market movements. For sentiment extraction, they employ a transformer-based model called FinBERT.

FinBERT-LSTM: Deep Learning based stock price prediction using News Sentiment Analysis

This is a relatively short article comparing an MLP and a simple LSTM with an LSTM incorporating daily news sentiment. It also provides an associated GitHub repository containing the code used in the study.

S&P 500 Index Using Mathematical-Based Sentiment Analysis and Deep Learning Models: A FinBERT Transformer Model and LSTM

This paper is more comprehensive, with greater emphasis on tuning the sentiment-informed LSTM. It offers an in-depth analysis of results across different parameters and time periods (e.g., the COVID-19 and Russia–Ukraine financial crises). However, the authors do not give access to their code.

Both papers use Yahoo Finance for stock market data, which seems easily accessible via the Yahoo API.

For sentiment analysis, they both use articles from the New York Times, however in different ways. The first does it by connecting to the nyt api, while the second uses web scraping tools to get the data (there are probably some limitations to the API use and even with the Brown premium subscription we aren't sure we can have full-access to the API).