Midas-Market

Home screen
Loading screen
Final product

About the Project

This project was born out of a desire to combine practicality with fun. I wanted to create an application that not only simplifies the process of comparing grocery prices from multiple stores but also provides a playful experience while you wait for the data to load.

What Inspired Me

The inspiration came from everyday frustrations with online grocery shopping—navigating different websites, searching for the best prices, and comparing products could be tedious. I thought, why not add a bit of fun to the process? Merging a simple game with real-time price scraping made the project uniquely engaging.

What I Learned

Web Scraping & Automation: I dove deep into Selenium to automate web interactions, extract real-time data, and handle dynamic content across different grocery websites.
Asynchronous Processing: Implementing background threads in Flask for scraping while updating a progress bar in real time was a major learning curve.
Frontend-Backend Integration: Integrating a playful game built with JavaScript (using a canvas for the interactive game) with a Flask backend taught me a lot about client-server communication.
Error Handling & Debugging: Overcoming challenges like handling GPU warnings in headless browsers and ensuring robust error handling was a significant part of the journey.

How I Built It

Backend Setup: I used Flask as the web framework to handle user requests, manage background scraping tasks, and serve the results. Selenium was used for web scraping, running in a headless Chrome instance.
Progress Tracking: A global progress variable and AJAX polling were implemented so that the client could receive real-time updates about the scraping process.
Interactive Game: While the scraping is in progress, users are entertained with a simple game where they control a basket to catch falling grocery items. The game uses an HTML canvas with JavaScript for real-time animations.
Data Aggregation & Export: Once scraping is complete, the results are aggregated, displayed in a user-friendly format, and can be downloaded as an Excel file for further analysis.

Challenges Faced

Handling Asynchronous Tasks: Ensuring that the background scraping did not block the main thread while updating the progress bar in real time required careful planning and debugging.
Dynamic Content: Scraping dynamic websites that load content asynchronously posed challenges in ensuring that the right elements were available before extraction.
Integration of Game and Progress Bar: Merging the interactive game with the real-time progress tracking was tricky. Balancing the user experience between entertaining gameplay and meaningful progress feedback was a key focus.
Environment Issues: Running headless browsers sometimes resulted in GPU-related warnings, which, although non-fatal, needed to be understood and managed.

Overall, this project was a rewarding blend of practical functionality and creative problem-solving, pushing me to expand my skills in both backend automation and frontend interactivity.