Wanting to build something fun and interesting that many people could use. All three of us are interesting in playing the stock market and thought we could use our programming knowledge to do it! This is a first start to help beginners to the play around with the market.
What it does
It displays historical data for about 5,000 stock symbols going back to August of last year. A web crawler was made to gather articles about these companies. Once the data was collected from the crawler, we used a basic form of NLP to determine if news article is positive or negative. Using a bag of words algorithm, we try to predict the fluctuation of the stock over an arbitrary amount of time.
How we built it
We used Go to connect to our Postgresql database that had historic stock data. We created an API that our web front end could pick up and graph that data using d3.js. Then we built a web crawler to find articles for companies to use as training data for our bag of words algorithm. The web front end was made in Go/HTML/CSS/JS and was using REST to connect to the API backend. We hosted the database on Digital Ocean to make it easier for all of us to get the data.
Challenges we ran into
Twitter's typeahead.js gave us some trouble with using Bootstrap 3. One of our biggest problems was waiting for our algorithms to train on the data we have. Other challenges are random JS/CSS bugs that made things difficult to debug. Being able to parse the articles and only get what is needed was really hard. Getting errors in our data made us realize we had problems with our NLP.
Accomplishments that we're proud of
We really like the look of the site. It is very simple and clean. This was the first time any of us had used d3.js. It was really awesome to work with and made some beautiful graphs. This is also the first time of doing anything with NLP and bag of words. We managed to get at least some data from that!
What we learned
We learned that d3.js is pretty cool and so is NLP! Some of the members of the group were not very familiar with JS and they learned a lot about it. Other members of the group have never used Go and this was their first time with it.
What's next for Stocks
We need better prediction models such as information extraction for our data. Upgrading from basic words to phrases using sentiment analysis. Graphing more of the behind the scenes data would be really nice too.