Scryer

Inspiration

As a CS student, I'm constantly amazed by the power of data and AI. However, the process of collecting and analyzing it can be tedious, especially for non-programmers. I've seen friends struggle with scraping websites for product data or manually combing through reviews for market research. Scryer was born from the desire to democratize data analysis, making it accessible and efficient for everyone.

What it does

Scryer is a web app that empowers users to extract targeted data from websites and unlock actionable insights. It uses web scraping with Gemini's powerful NLP capabilities to analyze extracted text data. Users define what website they need insights from and the information they need and Scryer automates the process, delivering clear, concise reports.

How I built it

• Backend (Node.js, Express.js): I opted for Node.js and Express.js to build the backend for Scryer.
• Frontend (Next.js): To deliver a smooth and dynamic user experience, I leveraged Next.js for the front end.
• Data Scraping (Puppeteer, Custom Algorithms): I utilized Puppeteer, a powerful headless browser library, to scrape text data and I created a custom algorithm for crawling. This approach provided more control over the process and allowed me to capture the internal linking structure of website being analyzed, offering a deeper understanding of the information architecture.

Challenges I ran into

One of the biggest hurdles was ensuring Scryer's crawler wouldn't get lost in the vastness of the internet. I implemented safeguards to prevent it from following links to different domains and meticulously tracked visited URLs to avoid duplicates.

Accomplishments that I'm proud of

I was able to integrate web scraping with Gemini's NLP to create a powerful data analysis pipeline accessible through a user-friendly interface.

What I learned

• The Importance of Flexibility: Data comes in all shapes and sizes. Building adaptable tools is crucial for handling the complexities of the web.
• The Power of AI for NLP: Gemini's capabilities were a game-changer, revealing hidden patterns and insights within the scraped data.
• User-centered Design is Key: A well-designed interface makes powerful tools accessible to a broader audience.

What's next for Scryer

• Advanced Data Visualization: Interactive charts and graphs could further enhance the user experience and make insights even clearer.
• Support for More Data Types: Expanding Scryer to handle images, videos, and social media data would broaden its potential applications.

Conclusion

Building Scryer has not only enhanced my technical skills but also taught me the value of persistence and innovative thinking in solving real-world problems. I'm excited about the potential of Scryer to transform the way people access and understand information online. Scryer can empower individuals, businesses, and researchers to make data-driven decisions and unlock a new era of web-based discovery.

Built With

• express.js
• mongodb
• next.js
• node.js
• puppeteer