> ## Documentation Index
> Fetch the complete documentation index at: https://docs.surfsky.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrapy

> Route Scrapy requests through a pool of Surfsky browsers.

Surfsky's Scrapy integrations replace the download handler: each request is fetched by a cloud browser from a pool and comes back as an ordinary Scrapy response for your spider to parse.

| Package                           | Use it for                                             |
| --------------------------------- | ------------------------------------------------------ |
| `scrapy-cloud-browser`            | Browser-backed downloads with ordinary Scrapy parsing. |
| `scrapy-playwright-cloud-browser` | Playwright page actions before parsing the response.   |

<span id="using-surfsky-with-scrapy" />

## Prerequisites

Set your [API token and base URL](/quickstart#before-you-start). Both integrations require at least one proxy URL; unlike a direct API request, the account's default pool is not used:

```bash theme={null}
export SURFSKY_PROXY_URL="http://user:pass@proxy.example.com:8080"
```

Create a Scrapy project if you do not have one:

```bash theme={null}
pip install scrapy
scrapy startproject surfsky_spider
cd surfsky_spider
```

<Tabs>
  <Tab title="scrapy-cloud-browser">
    ### Installation

    ```bash theme={null}
    pip install scrapy-cloud-browser
    ```

    ### Configuration

    Add to `surfsky_spider/settings.py`:

    ```python theme={null}
    import os

    TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
    EXTENSIONS = {"scrapy_cloud_browser.CloudBrowserExtension": 500}
    DOWNLOAD_HANDLERS = {
        "http": "scrapy_cloud_browser.CloudBrowserHandler",
        "https": "scrapy_cloud_browser.CloudBrowserHandler",
    }
    CLOUD_BROWSER = {
        "API_HOST": os.environ["SURFSKY_API_BASE_URL"],
        "API_TOKEN": os.environ["SURFSKY_API_TOKEN"],
        "PROXIES": [os.environ["SURFSKY_PROXY_URL"]],
        "NUM_BROWSERS": 1,
        "PAGES_PER_BROWSER": 100,
        "START_SEMAPHORES": 1,
        "PROXY_ORDERING": "round-robin",
    }
    ```

    ### Example spider

    Save as `surfsky_spider/spiders/example.py`:

    ```python theme={null}
    import scrapy

    class ExampleSpider(scrapy.Spider):
        name = "example"
        start_urls = ["https://example.com"]

        def parse(self, response):
            yield {"title": response.css("title::text").get()}
    ```

    This integration does not expose `BROWSER_SETTINGS` or `FINGERPRINT` through `CLOUD_BROWSER`. Use the Playwright integration below if you need those settings.
  </Tab>

  <Tab title="scrapy-playwright-cloud-browser">
    <span id="installation-1" />

    ### Installation

    ```bash theme={null}
    pip install scrapy-playwright-cloud-browser
    ```

    <span id="configuration-1" />

    ### Configuration

    Add to `surfsky_spider/settings.py`:

    ```python theme={null}
    import os

    TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
    EXTENSIONS = {"scrapy_playwright_cloud_browser.CloudBrowserExtension": 500}
    DOWNLOAD_HANDLERS = {
        "http": "scrapy_playwright_cloud_browser.CloudBrowserHandler",
        "https": "scrapy_playwright_cloud_browser.CloudBrowserHandler",
    }
    CLOUD_BROWSER = {
        "API_HOST": os.environ["SURFSKY_API_BASE_URL"],
        "API_TOKEN": os.environ["SURFSKY_API_TOKEN"],
        "PROXIES": [os.environ["SURFSKY_PROXY_URL"]],
        "NUM_BROWSERS": 1,
        "PAGES_PER_BROWSER": 100,
        "START_SEMAPHORES": 1,
        "PROXY_ORDERING": "round-robin",
        "BROWSER_SETTINGS": {"inactive_kill_timeout": 60},
        "FINGERPRINT": {"os": "win"},
    }
    ```

    <span id="example-spider-with-playwright-features" />

    ### Example spider with page actions

    Save as `surfsky_spider/spiders/example.py`. Use Scrapy 2.13 or later for `async start()`:

    ```python theme={null}
    import scrapy
    from scrapy_playwright.page import PageMethod

    class ExampleSpider(scrapy.Spider):
        name = "example"

        async def start(self):
            yield scrapy.Request(
                "https://example.com",
                meta={
                    "playwright": True,
                    "playwright_page_methods": [PageMethod("wait_for_selector", "h1")],
                },
            )

        def parse(self, response):
            yield {"title": response.css("title::text").get()}
    ```

    The integration creates a browser context for its requests. Do not use it to restore the default context's persistent profile storage; use [Playwright directly](/quickstart/playwright) for that workflow.
  </Tab>
</Tabs>

## Run the example

From the project directory:

```bash theme={null}
scrapy crawl example -O results.json
```

`results.json` should contain an item with `"title": "Example Domain"`.

## Pool settings

| Setting             | Default    | Purpose                                                              |
| ------------------- | ---------- | -------------------------------------------------------------------- |
| `NUM_BROWSERS`      | `1`        | Browser workers. Keep this within your account's available capacity. |
| `PAGES_PER_BROWSER` | `100`      | Number of requests processed before recycling a browser.             |
| `START_SEMAPHORES`  | `10`       | Maximum concurrent startups within a handler.                        |
| `PROXY_ORDERING`    | `"random"` | Select proxies randomly or use `"round-robin"`.                      |

A browser serves several requests before it is recycled, so do not assume each response comes from a fresh session. If startup repeats without producing responses, check the API host, token, and proxy URL, then [limits](/limits). Start with one browser until the spider works.

## Stop the sessions

The pool closes its browsers when the crawl finishes. After an interrupted crawl, list [active sessions](/sessions#find-running-sessions) and stop any browser left running, or stop them all:

```bash theme={null}
curl --fail-with-body -X POST \
  "$SURFSKY_API_BASE_URL/profiles/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"
```

Otherwise each browser stops after its inactivity timeout, 30 seconds by default.