Dynamic Website Articles Retriever

Extract. Organize. Inform.

Comment

Inspiration

This idea for this project started from the frustration of having to manually download and browse endless files for a statistic project where a huge amount of data is needed to be extracted. I believe anyone in the same situation would feel a certain level of frustration and nerve. The goal was clear: build a solution that bridges the gap between raw information and actionable insights.

What it does

This program is a very straight-forward and user-friendly tool that automates the process of collecting articles from dynamic websites, such as TechCrunch. Through its interface, the user can:

  • Specify the number of articles to scrape
  • Set a pause duration for flexible scraping time
  • View scraped articles in real-time
  • Have the articles automatically saved and organized into their own category This tool is simplifies and fastens the process of gathering head news, making it more readily accessible for journalists, researchers, and enthusiasts.

How we built it

  • HtmlUnit Library: This simulates web interactions to scrape dynamic content efficiently. Using XPath queries, we extracted article titles, categories, URLs and content from the website's pages.
  • Java Swing: To make a clean and intuitive GUI. Users can input parameters to tailor to their needs.
  • The backend logic runs in a separate thread to ensure the GUI remains responsive during scraping. Articles are saved in HTML files in categorized folders for easy access.

Challenges we ran into

  • Dynamic Website Structure: Frequently updated websites like TechCrunch can often change their HTML structure, and this requires adaptability in our scraping logic.
  • Concurrency Management: Ensuring the GUI remained responsive while handling large-scale scraping required careful implementation of multi-threading.

Accomplishments that we're proud of

  • Scalability: This tool can be adapted to scrape other websites or integrate with databases for larger-scale application.
  • File Organization: Articles are automatically categorized into folders based on their provided topics - making it easier for users to navigate their large database.

AI Involvement

  • Perplexity AI was used to give out a manual for the GUI and also helped enhanced the multi-threading aspect of the project.

Built With

Share this project:

Updates

Submission history