About
While there are loads of powerful commercial tools to deal with ESG investing, our web application, SusInvest, aims to give retail and individual investors a quick estimate of how environmentally, socially and governance-conscious a company is before doing a deep dive into ESG analysis.
Frameworks
Our web application is built using the Django framework with Python. It makes extensive use of Scrapy spiders to do web scraping, and the GDELT article database to do an initial filtering of articles.
How it works
Users input a query string (ideally a company name). The website then filters through the enormous GDELT website (for the past three months) to find all articles that mention the company. Scrapy was then used to crawl every website returned. The number of hits of certain words were counted and weights them according to a corpus of words (eg. "sustainable": 0.1). The final score is fed into a sigmoid function and displayed on the website as a final score.
Challenges
Implementing Scrapy as a Django script instead of as a command in the CLI proved challenging as most of the documentation was for the latter use of Scrapy. Furthermore, the back-end of Scrapy was not made to run multiple times in the same kernel, so workarounds were implemented. Lastly, the implementation and debugging of the Django-Scrapy integration proved much more difficult than expected.
Log in or sign up for Devpost to join the conversation.