Inspiration
After finding out that the hackathon was based on charity and humanitarian aid (at least one track), I decided to pursue bettering charity organizations. Shockingly, Forbes predicts that the majority of charities will fail 10 years from now, and numerous studies have shown that over 20% of charities fail in a few years. This is largely due to the lack of people knowing about the charities and the charities not having an established website, social media, etc. In order to minimize this, I decided to develop a scraper that could learn about every charity in the US and develop a comprehensive and easy to view report. For instance, the cause of the charity, the amount of money the charity has, and contact information of the board of trustees.
What it does
This program mines data dynamically. It requires an input of a newline delimited list of text of charity organization names and it returns a comprehensive report of each charity.
How we built it
I built this using Python, OpenCV, sci-kit learn, NumPy, Pandas, DigitalOcean, Flask, HTML/CSS, and other less integral libraries. Firstly, I decided on using the IRS database and scraping it for information on every charity in the US. I was able to successfully extract about 1.2 million nodes of information with labels of Name, EIN, City, State, and tax deduction code. Afterwards, I used Selenium and url manipulation to go to each individual charity site on IRS and download their PDF using Wget.
Challenges we ran into
The program uses data in bulk and I don't have a cloud provider so it was difficult to streamline the data mining process. I ultimately transitioned from data mining to creating the program without the database so that people with a cloud can create the database and connect it to their website.
Accomplishments that we're proud of
Utilizing numerous high level libraries including OpenCV, Selenium, NumPy, and Pandas. Creating a unique algorithm that can detect lines in PDF files.
What we learned
How to use OpenCV, Pandas, and Selenium in order to scrape and extract information from IRS PDFs.
What's next for Charlotte Charity Scraper
Run the program on a big scale and connect it with the front-end website, will complete a database.
Log in or sign up for Devpost to join the conversation.