Create a Web Scraping Application that helps your Community

Inspiration

MLH - LHD

What it does

Its a Web Scraping Application which scraps the Movies Data from IMDB site

How we built it

With Help of Node.js and we install request-promise , request , cheerio and json2csv so we can get our data into csv format. First , we get on IMDB site and inspect some basic data like title, summary, releasedate and rating.

const request = require("request-promise");
const cheerio = require("cheerio");
const fs = require("fs");
const json2csv = require("json2csv").Parser;

(async() => {
    let imdbData = []
    for(let movie of movies){
        const response = await request({
            uri : movie,
            headers : {
                accept: 
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
                "accept-encoding": "gzip, deflate, br",
                "accept-language": "en-US,en;q=0.9"
            },
            gzip : true
        });

        let $ = cheerio.load(response)
        let  title = $('div[class="title_wrapper"] > h1').text().trim()
        let rating = $('div[class="ratingValue"] > strong > span').text()
        let summary = $('div[class="summary_text"]').text().trim()
        let release_date = $('a[title="See more release dates"]').text().trim()

        imdbData.push({
            title,
            rating,
            summary,
            release_date
        });
    }
    const j2cp =new json2csv();
    const csv = j2cp.parse(imdbData);

    fs.writeFileSync("./imdb.csv",csv,"utf-8");
}

)();

And we create the array of movies link to get the data of multiple movies :)

What we learned

How we can scrap data from any website until its just for education purpose

What's next for Create a Web Scraping Application that helps your Community

Try to dive deep into web scraping

Built With

• github
• node.js