Scraping API

The Scraping API loads pages in a Surfsky browser over plain HTTP and returns the rendered HTML. It also waits for content, takes screenshots, and solves CAPTCHAs, with no framework connection to hold open. Use it when you need HTML or a screenshot and do not want to hold a WebSocket connection. For forms, multiple tabs, or click sequences, use a framework or an SDK. To have a coding agent set this up in your project, give it the setup prompt.

For clicks, typing, and scrolling, use human-emulation commands through the SDK or CDP. The Scraping API’s human_actions option only performs random browsing actions; it does not accept a custom sequence.

Scraping more than one page? Send the URLs as a batch to one browser instead of starting a browser per page.

Prerequisites

Set your API token and base URL. The examples start a browser with an empty request body, which uses your account’s default proxy pool. Proxy, fingerprint, and other start options are in the API reference.

Run an example

The examples use a one-time profile. To reuse a saved login, pass a persistent profile UUID to the session call instead.

curl
Python (async)
JavaScript

INTERNAL_UUID=$(curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/one_time" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" -d '{}' | jq -r .internal_uuid)

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "wait_for": "h1"}'

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

Requires Python 3.12 or newer.

pip install surfsky

Save as scrape.py:

import asyncio
from surfsky import AsyncSurfsky


async def main():
    async with AsyncSurfsky() as client, client.session() as session:
        result = await client.profiles.scrape(
            session, "https://example.com", wait_for="h1"
        )
        print(result.status)
        print(result.content)


asyncio.run(main())

Run it:

python scrape.py

Requires Node.js 24 or newer, or Bun, for await using.

npm install surfsky

Save as scrape.mjs:

import { Surfsky } from "surfsky";

const client = new Surfsky();
await using session = await client.session();
const result = await client.profiles.scrape(session, "https://example.com", {
  wait_for: "h1",
});
console.log(result.status);
console.log(result.content);

Run it:

node scrape.mjs

The SDK examples print 200 followed by HTML containing Example Domain. The curl example returns the JSON response described below.

Click, type, and scroll

For a specific input sequence, use client.browser() instead of the client.session() and profiles.scrape() workflow. Its input methods call Surfsky’s Human.* primitives. After creating a client, run this on your target form, replacing the URL and selector:

async with client.browser() as browser:
    await browser.goto("https://your-site.example/form")
    await browser.click("input[name='email']")
    await browser.type("input[name='email']", "[email protected]")
    await browser.scroll(delta_y=600, duration=1.2)

await using browser = await client.browser();
await browser.goto("https://your-site.example/form");
await browser.click("input[name='email']");
await browser.type("input[name='email']", "[email protected]");
await browser.scroll({ deltaY: 600, duration: 1.2 });

The SDK stops the browser when the block or scope exits. Use this as a separate workflow: the HTTP scraper takes over the page and closes extra tabs, so do not combine it with CDP actions in the same browser. See Human emulation for the underlying commands.

Request parameters

POST /profiles/{internal_uuid}/scrape accepts:

Field	Default	Meaning
`url`	Required	Page to visit.
`wait_until`	`"domcontentloaded"`	Navigation condition: `commit`, `domcontentloaded`, `load`, or `networkidle`.
`wait_for`	Omitted	CSS or XPath selector to wait for after navigation.
`timeout`	`30000`	Navigation and selector timeout, in milliseconds.
`wait`	`0`	Extra delay after loading, in seconds, up to `60`.
`screenshot`	`false`	Include a Base64 PNG screenshot.
`auto_captcha_solve`	`false`	Solve CAPTCHAs during the scrape. Requires `anti_captcha` at browser start.
`human_actions`	`0`	Random browsing actions before returning, up to `3`.

wait_for with a selector for the content you need is more reliable than networkidle, which can time out on pages with background traffic. The gateway gives each /scrape call 120 seconds in total, covering every page in a batch. timeout and wait apply inside that budget.

Read the response

{
  "success": true,
  "msg": "",
  "data": {
    "url": "https://example.com/",
    "status": 200,
    "status_text": "OK",
    "content": "<!doctype html><html>...</html>",
    "cookies": [],
    "screenshot": "iVBORw0KGgo..."
  }
}

success reports whether the Surfsky request succeeded. data.status is the target site’s HTTP status, so a 200 from Surfsky can carry a 403 or a challenge page from the site. Check the target status and content. screenshot is present only when requested; decode it with Base64 to get a PNG.

Scrape several pages

Whenever your job allows it, send several URLs to one browser instead of starting a browser per page. Each start costs time and a browser slot, and the pages in a batch share the same cookies. Send them as an array of request objects to the same endpoint:

[
  {"url": "https://example.com", "wait_for": "h1"},
  {"url": "https://example.org", "screenshot": true}
]

Pages run one after another in request order, and data comes back as an array in that order; failed items carry error and status_code instead of a page. Keep a batch within the 120-second limit, and use several browsers when you need pages fetched in parallel.

Stop the session

The SDK stops the session when its scope exits, including on errors. For direct HTTP requests, stop the browser with the internal_uuid from the start response:

curl --fail-with-body -X POST \
  "$SURFSKY_API_BASE_URL/profiles/INTERNAL_UUID/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

Otherwise the browser stops after the inactivity timeout. Do not mix the Scraping API with CDP automation in the same browser; the scraper takes over the page and closes extra tabs.

Get started

Connect your tools

Browser sessions

Browser actions

CAPTCHA guides

Debugging

Performance and limits

About Surfsky

Prerequisites

Run an example

Click, type, and scroll

Request parameters

Read the response

Scrape several pages

Stop the session

​Prerequisites

​Run an example

​Click, type, and scroll

​Request parameters

​Read the response

​Scrape several pages

​Stop the session

Prerequisites

Run an example

Click, type, and scroll

Request parameters

Read the response

Scrape several pages

Stop the session