Inspiration
Data collection on stock list and prices
What it does
Goes to https://www.thestar.com.my/Business/Marketwatch/Stock-List/?alphabet=**X** to collect stock list. Then, it goes to each of the stock index https://www.thestar.com.my/Business/Marketwatch/Stocks?qcounter=**X** to collect stock data (board, stock code, open, high, low, last, change, change%, and volume)
How I built it
I used beautiful soup to crawl and pgAdmin to store data.
Challenges I ran into
The website coding structure keep changing, so the crawler must be keep updated. Crawling in dynamic table and dynamic page. To understand which data is important for data collection process Run the crawler everyday.
Accomplishments that I'm proud of
Completed within one week
What I learned
Understanding of stock data
What's next for Crawler: TheStar Stock-List
Data analysis on stock data
This project is under active development.
