How to Web Scrape movie data from Myuve, IMDb Using python

Web scrape movies

Web Scraping is the process of gathering useful information online and making meaningful insights from it. In a way, web scraping is automating the method of knowledge collection. If you’re scraping data for educational purposes or as a part of your job following all company policies, then it’s very unlikely of you to possess any problems. So, it’s advisable to try to do some research to make sure that you simply aren’t violating any Terms of Service.

Some companies like Google, Facebook, Twitter, etc., offer Application Programming Interfaces (APIs) that allow you to access their data during a predefined format like JSON, XML.

Movie data analysis using Python

Python offers a spread of libraries to scrape online like BeautifulSoup, Requests, Scrapy, Selenium. If you’re starting with web scraping, then Beautiful Soup is going to be the straightforward option. Also, if you’re building big projects, Beautiful Soup won’t be a wise option as they’re not flexible and are difficult to take care of because the project size increases.

Python IMDb web scrape movies

We can scrape the IMDb movie ratings and their details with the assistance of the BeautifulSoup library of Python.

Below is the list of modules required to scrape from IMDB.

1.) Requests:

The requests library is an integral neighborhood of Python for creating HTTP requests to a specified URL. Whether it’s REST APIs or Web Scraping, requests must be learned for proceeding further with these technologies. When one makes an invitation to a URI, it returns a response.

2.) html5lib:

A pure-python library for parsing HTML. it’s designed to evolve to the WHATWG HTML specification, as is implemented by all major web browsers.

3.) bs4:

BeautifulSoup object is provided by Beautiful Soup which may be a web scraping framework for Python. Web scraping is the process of extracting data from the website using automated tools to form the method faster.

Steps:

Steps to implement web scraping in python to extract IMDb movie ratings and their ratings.

1.) Import the specified modules.

2.) Access the HTML content from the webpage by assigning the URL and creating a soap object.

3.) Extract the movie ratings and their details. Here, we are extracting data from the BeautifulSoup objects using Html tags like href, title, etc.

movies = soup.select('td.titleColumn')
links = [a.attrs.get('href') for a in soup.select('td.titleColumn a')]
crew = [a.attrs.get('title') for a in soup.select('td.titleColumn a')]
ratings = [b.attrs.get('data-value') for b in soup.select('td.posterColumn span[name=ir]')]
votes = [b.attrs.get('data-value') for b in soup.select('td.ratingColumn strong')]

4.) After extracting the movie details, create an empty list and store the small print in a dictionary, then add them to an inventory.

5.) Now our list is crammed with top IMDB movies alongside their details. Finally, display the list of movie details:

For a movie on list:

print(movie['place'], '-', movie['movie_title'], '('+movie['year'] +

        ') -', 'Starring:', movie['star_cast'], movie['rating'])

How to extract data from IMDb:

Just attend the detailed page of any movie, open the Parsers extension and click on the data you’d wish to extract. you’ll select several sorts of data on the page, for example, title movies, duration, release date, Image URL, description. Just click Add New Field. Next Start data extraction within the Parsers extension.

IMDb-web scraping using GitHub

This can be used to crawl the IMDB website to scrape movies’ information then store the data in JSON format.

1.) Crawling

Clone the repo and navigate into the IMDB-Scraper folder.

2.) Create and activate a virtual environment.

3.)Install all dependencies.

4.) Navigate into the imdb_scraper folder.

5.) You can change the starting page of the crawler within the file by changing the “SEARCH_QUERY” variable.

Copy the generated URL and paste it in situ of default URL. By default:

SEARCH_QUERY = (
    'https://www.imdb.com/search/title?'
    'title_type=feature&'
    'user_rating=1.0,10.0&'
    'countries=us&'
    'languages=en&'
    'count=250&'
    'view=simple'
)

6.) Start the crawler.

7.) Data will be stored in a JSON file named movie.json located at

" IMDB-Scraper/imdb-scraper/data/movie.json"

The final data will be in the form:

[
    ...
    {
        ...
    },

    {
        "title": "12 Strong",
        "rating": "R",
        "year": "2018",
        "users_rating": "6.6",
        "votes": "42,919",
        "metascore": "54",
        "img_url": "https://m.media-amazon.com/images/M/MV5BNTEzMjk3NzkxMV5BMl5BanBnXkFtZTgwNjY2NDczNDM@._V1_UX182_CR0,0,182,268_AL__QL50.jpg",
        "countries": [
            "USA"
        ],
        "languages": [
            "English",
            "Dari",
            "Russian",
            "Spanish",
            "Uzbek"
        ],
        "actors": [
            "Chris Hemsworth",
            "Michael Shannon",
            "Michael Peña",
            "Navid Negahban",
            "Trevante Rhodes",
            "Geoff Stults",
            "Thad Luckinbill",
            "Austin Hébert",
            "Austin Stowell",
            "Ben O'Toole",
            "Kenneth Miller",
            "Kenny Sheard",
            "Jack Kesy",
            "Rob Riggle",
            "William Fichtner"
        ],
        "genre": [
            "Action",
            "Drama",
            "History",
            "War"
        ],
        "tagline": "The Declassified True Story of the Horse Soldiers",
        "description": "12 Strong tells the story of the first Special Forces team deployed to Afghanistan after 9/11; under the leadership of a new captain, the team must work with an Afghan warlord to take down the Taliban.",

        "directors": [
            "Nicolai Fuglsig"
        ],
        "runtime": "130 min",
        "imdb_url": "https://www.imdb.com/title/tt1413492/"
    },
    {
        ...
    }
    ...
]