API Overview

Scrapingpass allows you to scrape any website and extract data easily. It provides you with powerful web scraping and robotic process automation (RPA) tools to make your work easy for you. Either normal websites or modern client side rendered applications, scrapingpass allows you to scrape all of them. It takes care of rotating proxies, network issues with smart retries and scraping data itself using css selectors, etc.

Key Features

• Easy to setup and integrate with your flow.
• Supports JS rendering for modern web applications using client side rendering.
• Provides several proxy types.
• Supports various rendering and output formats.
• Proxy mode works like any other HTTP proxy which allows you to pass headers, etc. to remote easily.
• Many more options and configurations supported to power up your scraping tasks.
• Fine tuned for scraping purpose.
• Automate several aspects of the web.
• Allows you to manage and aggregate data from various websites into a single workflow and execute your project seamlessly.

Example API Usage

Scrapingpass API can be easily integrated into your new or existing project. As of now, Scrapingpass provides two interfaces to integrate and consume the API. Depending on your project setup, you may use either of these.

HTTP REST API

Scrapingpass provides really simple HTTP REST API without multiple confusing endpoints. A single endpoint for all your scraping needs. Each behaviour of parsing and response can be modified by passing options as query parameters.

cURL

curl 'https://api.scrapingpass.com?api_key=YOUR-API-KEY&url=https://example.com/'

Python

import requests

url = "https://api.scrapingpass.com"

options = {
    "api_key": "YOUR-API-KEY",
    "url": "https://example.com/"
}

response = requests.get(url, params=options)

tip

Get started and dive directly into the HTTP REST API documentation.

HTTP Proxy Mode

HTTP Proxy Mode is the most intuitive way to consume the API without making too much changes to your existing project. It can be incorporated into your ecosystem by simply injecting our proxy into your request methods. That's it. Scrapingpass will automatically do the rest for you.

Options and other stuffs like your API key is needed to be configured in this proxy URL as HTTP basic authorization in form of proxy's username and password.

cURL

curl --insecure \
    --proxy 'http://YOUR-API-KEY:scrapingpass&[email protected]:8080' \
    'https://example.com/'

Python

import requests

proxy_url = "http://YOUR-API-KEY:scrapingpass&[email protected]:8080"
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get("https://example.com/", proxies=proxies, verify=False)

tip

Get started and dive directly into the HTTP Proxy Mode documentation.

Getting Help

Send an email along with your problem description and Scrapingpass email to [email protected]. We will be glad to help you out.

Reporting Bugs

If you believe you're experiencing a bug with Scrapingpass API or want to report incorrect documentation, open an issue on its issue tracker.