ParseDoD

Contracts, Faster.

Comment 3

Inspiration

The inspiration for ParseDoD came from the need to efficiently analyze and visualize Department of Defense (DOD) contracts data. Current data is in a very dense paragraph based format, making it difficult to search and analyze. By automating the process of scraping through DOD contracts, extracting relevant purchase information, and presenting it in a user-friendly format, ParseDoD aims to provide valuable insights into government spending.

What it does

ParseDoD is a web application that utilizes a web scraper to gather data from DOD contracts. It is parsed through this data using GPT-3.5 to extract details about purchases in addition to heuristic detection of certain attributes, such as the items procured, the amounts spent, and the companies involved. Finally, ParseDoD presents this information through interactive graphs and charts on a web application, allowing users to explore and understand DOD spending patterns easily.

How we built it

  • Jsoup and Selenium for automated DoD contracts date range search and scraping
  • GPT-3.5 for extraction of text-based attributes (company name, location, etc)
  • Java heuristic detection for enumeration data (branch, dates, etc)
  • MongoDB storage of structured data
  • Java EE/Servlets web application backend
  • Bootstrap/HTML/CSS/JS frontend (mostly handwritten)
  • Google Maps API and Chart.JS API for insight visualization.
  • Manual data collection and verification of edge cases and randomly selected contracts to ensure accuracy: >99%.

Challenges we ran into

One of the primary challenges we encountered was efficiently scraping and parsing large volumes of DOD contracts data while ensuring accuracy and reliability, dealing with inconsistencies in the data structure, and the layout of the data presented in the contracts. Adapting the prompt to handle different formats posed significant challenges. Additionally, designing an intuitive user interface and optimizing performance were ongoing challenges throughout the development process.

Accomplishments that we're proud of

We're proud to have developed a robust web scraper capable of extracting valuable purchase information from DOD contracts reliably. Creating an intuitive and visually appealing user interface for presenting complex data in an understandable format was also a significant achievement. Moreover, overcoming technical challenges and refining the application to deliver a seamless user experience was immensely satisfying.

What we learned

Through building ParseDoD, we gained invaluable experience in web scraping, data parsing, web development, and data visualization. We learned how to navigate through the complexities of government contract data and extract meaningful insights efficiently. Additionally, we honed our skills in frontend and backend development, as well as project management and collaboration within a team setting.

What's next for ParseDoD

In the future, we envision expanding ParseDoD's capabilities by incorporating advanced analytics features, such as predictive modeling and trend analysis, to provide deeper insights into DOD spending patterns. Furthermore, we plan to enhance the scalability and performance of the application to handle even larger volumes of data. Additionally, we aim to integrate user feedback to continually improve the user experience and ensure ParseDoD remains a valuable tool for analyzing DOD contracts data.

Share this project:

Updates