Scrapy

Surfsky’s Scrapy integrations use a pool of cloud browsers to fetch pages. The download handler returns an ordinary Scrapy response for your spider to parse. To have a coding agent set this up in your project, give it the setup prompt.

Use human-emulation commands for clicks, typing, and scrolling. Keep Scrapy selectors and Playwright’s navigation and waiting for reading and preparing the page.

Package	Use it for
`scrapy-cloud-browser`	Browser-backed downloads with ordinary Scrapy parsing.
`scrapy-playwright-cloud-browser`	Playwright page actions before parsing the response.

Prerequisites

Set your API token and base URL. Both integrations require at least one proxy URL; unlike a direct API request, the account’s default pool is not used:

export SURFSKY_PROXY_URL="http://user:[email protected]:8080"

Create a Scrapy project if you do not have one:

pip install scrapy
scrapy startproject surfsky_spider
cd surfsky_spider

scrapy-cloud-browser
scrapy-playwright-cloud-browser

Installation

pip install scrapy-cloud-browser

Configuration

Add to surfsky_spider/settings.py:

import os

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
EXTENSIONS = {"scrapy_cloud_browser.CloudBrowserExtension": 500}
DOWNLOAD_HANDLERS = {
    "http": "scrapy_cloud_browser.CloudBrowserHandler",
    "https": "scrapy_cloud_browser.CloudBrowserHandler",
}
CLOUD_BROWSER = {
    "API_HOST": os.environ["SURFSKY_API_BASE_URL"],
    "API_TOKEN": os.environ["SURFSKY_API_TOKEN"],
    "PROXIES": [os.environ["SURFSKY_PROXY_URL"]],
    "NUM_BROWSERS": 1,
    "PAGES_PER_BROWSER": 100,
    "START_SEMAPHORES": 1,
    "PROXY_ORDERING": "round-robin",
}

Example spider

Save as surfsky_spider/spiders/example.py:

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    start_urls = ["https://example.com"]

    def parse(self, response):
        yield {"title": response.css("title::text").get()}

This integration does not expose BROWSER_SETTINGS or FINGERPRINT through CLOUD_BROWSER. Use the Playwright integration below if you need those settings.

Installation

pip install scrapy-playwright-cloud-browser

Configuration

Add to surfsky_spider/settings.py:

import os

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
EXTENSIONS = {"scrapy_playwright_cloud_browser.CloudBrowserExtension": 500}
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright_cloud_browser.CloudBrowserHandler",
    "https": "scrapy_playwright_cloud_browser.CloudBrowserHandler",
}
CLOUD_BROWSER = {
    "API_HOST": os.environ["SURFSKY_API_BASE_URL"],
    "API_TOKEN": os.environ["SURFSKY_API_TOKEN"],
    "PROXIES": [os.environ["SURFSKY_PROXY_URL"]],
    "NUM_BROWSERS": 1,
    "PAGES_PER_BROWSER": 100,
    "START_SEMAPHORES": 1,
    "PROXY_ORDERING": "round-robin",
    "BROWSER_SETTINGS": {"inactive_kill_timeout": 60},
    "FINGERPRINT": {"os": "win"},
}

Example spider with page actions

Save as surfsky_spider/spiders/example.py. The adapter uses Scrapy 2.12, so this spider defines start_requests(). Page actions can still be asynchronous.

import scrapy
from scrapy_playwright.page import PageMethod

class ExampleSpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={
                "playwright": True,
                "playwright_page_methods": [PageMethod("wait_for_selector", "h1")],
            },
        )

    def parse(self, response):
        yield {"title": response.css("title::text").get()}

The integration creates a browser context for its requests. Do not use it to restore the default context’s persistent profile storage; use Playwright directly for that workflow.

Run the example

From the project directory:

scrapy crawl example -O results.json

results.json should contain an item with "title": "Example Domain".

Click, type, and scroll

With scrapy-playwright-cloud-browser, pass an async callable to PageMethod to send page-level CDP commands before Scrapy receives the response:

async def fill_email(page):
    cdp = await page.context.new_cdp_session(page)
    try:
        await cdp.send("Human.click", {"selector": "input[name='email']"})
        await cdp.send("Human.type", {"text": "[email protected]"})
        await cdp.send("Human.scroll", {"deltaY": 600, "duration": 1.2})
    finally:
        await cdp.detach()

Define the function above the spider class, replace the request URL with your form page, and set "playwright_page_methods": [PageMethod(fill_email)] in its request metadata. Replace the selector with an input on that page. The scrapy-cloud-browser handler does not expose a Playwright page. Use the Playwright integration for these actions.

Use the Surfsky SDK

To manage a session directly from a spider, use the SDK with the Scraping API. This standalone example uses Scrapy to parse the returned HTML. It requires Python 3.12+ and Scrapy 2.13+. Use a separate environment from the Playwright cloud browser integration, which pins Scrapy 2.12:

python -m venv sdk-env
source sdk-env/bin/activate
pip install surfsky "scrapy>=2.13"

Save as sdk_spider.py outside your configured Scrapy project. This example does not use the cloud browser extensions or download handlers:

import scrapy
from surfsky import AsyncSurfsky


class SdkSpider(scrapy.Spider):
    name = "sdk_example"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    }

    async def start(self):
        async with AsyncSurfsky() as client, client.session() as session:
            result = await client.profiles.scrape(
                session, "https://example.com", wait_for="h1"
            )
            yield {"title": scrapy.Selector(text=result.content).css("title::text").get()}

Run it:

scrapy runspider sdk_spider.py -O results.json

The SDK stops the browser when the session scope exits, including on errors. It uses your account’s default proxy pool unless you pass proxy to client.session(), and a persistent profile UUID reuses saved state.

Pool settings

Setting	Default	Purpose
`NUM_BROWSERS`	`1`	Browser workers. Keep this within your account’s available capacity.
`PAGES_PER_BROWSER`	`100`	Number of requests processed before recycling a browser.
`START_SEMAPHORES`	`10`	Maximum concurrent startups within a handler.
`PROXY_ORDERING`	`"random"`	Select proxies randomly or use `"round-robin"`.

A browser serves several requests before it is recycled, so do not assume each response comes from a fresh session. If startup repeats without producing responses, check the API host, token, and proxy URL, then limits. Start with one browser until the spider works.

Stop the sessions

The pool closes its browsers when the crawl finishes. After an interrupted crawl, list active sessions and stop any browser left running, or stop them all:

curl --fail-with-body -X POST \
  "$SURFSKY_API_BASE_URL/profiles/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

Otherwise each browser stops after its inactivity timeout, 30 seconds by default.

Get started

Connect your tools

Browser sessions

Browser actions

CAPTCHA guides

Debugging

Performance and limits

About Surfsky

Prerequisites

Installation

Configuration

Example spider

Installation

Configuration

Example spider with page actions

Run the example

Click, type, and scroll

Use the Surfsky SDK

Pool settings

Stop the sessions

​Prerequisites

​Installation

​Configuration

​Example spider

​Installation

​Configuration

​Example spider with page actions

​Run the example

​Click, type, and scroll

​Use the Surfsky SDK

​Pool settings

​Stop the sessions

Prerequisites

Installation

Configuration

Example spider

Installation

Configuration

Example spider with page actions

Run the example

Click, type, and scroll

Use the Surfsky SDK

Pool settings

Stop the sessions