Project Melvin Doo

Inspiration

We were inspired by the social and governance themes of this Hackathon. We wanted to provide a tool that would allow the public to view voting records, as well as stock trades, of congresspeople in the United States to find indications of conflicts of interest between voting behaviour and trades, and insider trading.

What it does

The tool is a webapp that allows users to enter the name of a congressperson, and retrieve both voting records and stock trades for that congressperson. The user is then able to select particular categories (i.e. health, technology), and filter the results to produce only voting records and stock trades related to those categories. Furthermore, the user can view the historical price of individual stocks, and are able to see at what point congresspeople made particular trades.

How we built it

The backend and data processing are built entirely in Python. The web backend utilises Flask, and the data processing components make heavy use of scraping tools like Selenium, BeautifulSoup, and requests. There is a MySQL server running that stores bill votes and stock trades, and historical stock price data is pulled at runtime from a script that interacts with Google Sheets, to make use of the =GOOGLEFINANCE function, which allows access to the publicly deprecated Google Finance API.

Challenges we ran into

There were some challenges with bill and company categorisation, that make it difficult to determine exact conflicts of interest. There were also issues integrating the database, especially due to version differences between Windows and Linux versions of MySQL.

Accomplishments that we're proud of

We were able to scrape and process significant amounts of data, as well as eventually get our bill categorisation working. Furthermore, our novel approach to getting free access to a finance API is something we are proud of.

What we learned

We learned how to plan a large project entirely from the beginning, as well as deal with unexpected difficulties along the way. We also learned a lot about specific technologies, like MySQL, and nltk.

What's next for Project Melvin Doo

There is plenty of room for feature growth within the project, such as more sophisticated analysis of stock prices and corresponding trades/bill votes, as well as potential to improve the front-end and user experience of our project.

Built With

Submitted to

Hack Cambridge Atlas
- Winner Hackathons UK - Hackiest Hack

Created by

I worked on data aggregation and processing, including retrieving voting data and stock market trades made by members of Congress. This involved setting up the Google Finance API, which required a substantial amount of hoop-jumping to set up.

Following this, I worked on the backend API, as well as data visualisation for the webapp. This allows users to view lists of votes with potential conflicts of interests, displaying any trades that may have influenced the decision.

Maxim Webb
My most notable contribution to the project was my work on setting up the MySQL database and providing an API for it that could be used by others on the team. I worked on setting up the calls to automatically populate the database using the scraped data provided, as well as creating the queries to return useful information for the front end.

William Hayter-Dalgliesh
I worked on various parts of the project: scraping trade data of congresspeople, as well as setting up the web front and backend initially. I later worked on populating our database with the data we had scraped over the project. I also worked on some final links between the front end and back end in creating some endpoints and writing queries.

Joe Cheng
I worked on finding a suitable model to classify bills into pre-defined categories (based on traded companies). I began by looking into using a Latent Dirichlet Allocation model, however this does not allow pre-defined categories to be set and the categories it formed weren't close enough to our required categories.

In the end, a simple keyword classification was used to categories the bills.

an578