Scrapy

Surfsky’s Scrapy integrations replace the download handler: each request is fetched by a cloud browser from a pool and comes back as an ordinary Scrapy response for your spider to parse.

Package	Use it for
`scrapy-cloud-browser`	Browser-backed downloads with ordinary Scrapy parsing.
`scrapy-playwright-cloud-browser`	Playwright page actions before parsing the response.

Prerequisites

Set your API token and base URL. Both integrations require at least one proxy URL; unlike a direct API request, the account’s default pool is not used:

export SURFSKY_PROXY_URL="http://user:[email protected]:8080"

Create a Scrapy project if you do not have one:

pip install scrapy
scrapy startproject surfsky_spider
cd surfsky_spider

scrapy-cloud-browser
scrapy-playwright-cloud-browser

Installation

pip install scrapy-cloud-browser

Configuration

Add to surfsky_spider/settings.py:

import os

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
EXTENSIONS = {"scrapy_cloud_browser.CloudBrowserExtension": 500}
DOWNLOAD_HANDLERS = {
    "http": "scrapy_cloud_browser.CloudBrowserHandler",
    "https": "scrapy_cloud_browser.CloudBrowserHandler",
}
CLOUD_BROWSER = {
    "API_HOST": os.environ["SURFSKY_API_BASE_URL"],
    "API_TOKEN": os.environ["SURFSKY_API_TOKEN"],
    "PROXIES": [os.environ["SURFSKY_PROXY_URL"]],
    "NUM_BROWSERS": 1,
    "PAGES_PER_BROWSER": 100,
    "START_SEMAPHORES": 1,
    "PROXY_ORDERING": "round-robin",
}

Example spider

Save as surfsky_spider/spiders/example.py:

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    start_urls = ["https://example.com"]

    def parse(self, response):
        yield {"title": response.css("title::text").get()}

This integration does not expose BROWSER_SETTINGS or FINGERPRINT through CLOUD_BROWSER. Use the Playwright integration below if you need those settings.

Installation

pip install scrapy-playwright-cloud-browser

Configuration

Add to surfsky_spider/settings.py:

import os

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
EXTENSIONS = {"scrapy_playwright_cloud_browser.CloudBrowserExtension": 500}
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright_cloud_browser.CloudBrowserHandler",
    "https": "scrapy_playwright_cloud_browser.CloudBrowserHandler",
}
CLOUD_BROWSER = {
    "API_HOST": os.environ["SURFSKY_API_BASE_URL"],
    "API_TOKEN": os.environ["SURFSKY_API_TOKEN"],
    "PROXIES": [os.environ["SURFSKY_PROXY_URL"]],
    "NUM_BROWSERS": 1,
    "PAGES_PER_BROWSER": 100,
    "START_SEMAPHORES": 1,
    "PROXY_ORDERING": "round-robin",
    "BROWSER_SETTINGS": {"inactive_kill_timeout": 60},
    "FINGERPRINT": {"os": "win"},
}

Example spider with page actions

Save as surfsky_spider/spiders/example.py. Use Scrapy 2.13 or later for async start():

import scrapy
from scrapy_playwright.page import PageMethod

class ExampleSpider(scrapy.Spider):
    name = "example"

    async def start(self):
        yield scrapy.Request(
            "https://example.com",
            meta={
                "playwright": True,
                "playwright_page_methods": [PageMethod("wait_for_selector", "h1")],
            },
        )

    def parse(self, response):
        yield {"title": response.css("title::text").get()}

The integration creates a browser context for its requests. Do not use it to restore the default context’s persistent profile storage; use Playwright directly for that workflow.

Run the example

From the project directory:

scrapy crawl example -O results.json

results.json should contain an item with "title": "Example Domain".

Pool settings

Setting	Default	Purpose
`NUM_BROWSERS`	`1`	Browser workers. Keep this within your account’s available capacity.
`PAGES_PER_BROWSER`	`100`	Number of requests processed before recycling a browser.
`START_SEMAPHORES`	`10`	Maximum concurrent startups within a handler.
`PROXY_ORDERING`	`"random"`	Select proxies randomly or use `"round-robin"`.

A browser serves several requests before it is recycled, so do not assume each response comes from a fresh session. If startup repeats without producing responses, check the API host, token, and proxy URL, then limits. Start with one browser until the spider works.

Stop the sessions

The pool closes its browsers when the crawl finishes. After an interrupted crawl, list active sessions and stop any browser left running, or stop them all:

curl --fail-with-body -X POST \
  "$SURFSKY_API_BASE_URL/profiles/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

Otherwise each browser stops after its inactivity timeout, 30 seconds by default.

Get started

Connect your tools

Browser sessions

Browser actions

CAPTCHA guides

Debugging

Performance and limits

About Surfsky

Prerequisites

Installation

Configuration

Example spider

Installation

Configuration

Example spider with page actions

Run the example

Pool settings

Stop the sessions

​Prerequisites

​Installation

​Configuration

​Example spider

​Installation

​Configuration

​Example spider with page actions

​Run the example

​Pool settings

​Stop the sessions

Prerequisites

Installation

Configuration

Example spider

Installation

Configuration

Example spider with page actions

Run the example

Pool settings

Stop the sessions