Machine Learning Stock Prediction and Sentiment Analysis

• 

  LSTM Learning from APPL Stock technical indicators

• 

  Sentiment Analysis Score for Headlines

Inspiration

Using machine learning and AI to improve stock market predictions. By using ML to predict stock price and sentiment analysis from web scraping, a better buy or sell decision can be made.

What it does

The dataset is augmented with several technical indicators (RSI, exponential moving average, rate of change) that are computed using the team's code. An LSTM is trained on the new data and validation curves are used to verify that the model is successfully learning from the data. A reddit web scraper was also implemented that uses the VADER sentiment score on a given headline.

How we built it

First a dataset from Kaggle was taken, and the technical indicator computations performed and added to the dataset. The LSTM was then trained to predict the next time step in the sequence. Loss curves for training and test losses were plotted to verify the model was learning. For the sentiment analysis, a Reddit web scraper was implemented. A FinGPT model was loaded to provide the VADER sentiment score.

Challenges we ran into

One challenge with the LSTM section was successfully lining up the tensor dimensions to compute the losses. Also, modifying a fairly large starting code to work on a dataset with many more columns was tricky. In addition, debugging why the losses were so high and the model wasn't learning was difficult. For sentiment analysis, it was difficult to get historical social media data to train.

Accomplishments that we're proud of

We are proud of our ideation of the project that how we could leverage years of knowledge of pure mathematical technical indicators with something that can be vague as news and train our model on both to learn about both a long and a short term market strategy. The learning got from technical indicators will always be relevant from a long term market strategy perspective but the relevance of news in the short term strategy is intended to make the model powerful.

What we learned

We learned how technical indicators which are built using years of knowledge can be leveraged along with the historical price trends to make the analysis and machine learning even stronger. We also learnt that some sprinkling of sentiment analysis can help you save yourself from negative sentiments and news that might hamper price trends in the short term.

What's next for Machine Learning Stock Prediction and Sentiment Analysis

As next steps we want to take this forward as a full blown application where users can select stocks and get predictions. We would also like to study the impact of various technical indicators and research the capabilities of what works in what conditions. On the other hand we would also like to involve data from more social media platforms to be able to make decisions quickly. We could also have users be able to tune the weightage of news vs indicators so they can personalise their own investment strategies.

Built With

• matplotlib
• nltk
• numpy
• pandas
• pandas-ta
• praw
• python
• pytorch
• tensorflow
• transformers