Scraping API

The Scraping API accepts HTTP requests to load pages in a Surfsky browser and return rendered HTML. It supports waiting for content, screenshots, CAPTCHA solving, and human input emulation without a framework connection. Use it when you need HTML or a screenshot and do not want to hold a WebSocket connection. For forms, multiple tabs, or click sequences, use a framework or an SDK.

Prerequisites

Set your API token and base URL. The examples start a browser with an empty request body, which uses your account’s default proxy pool. Proxy, fingerprint, and other start options are in the API reference.

Run an example

The example starts a one-time browser, scrapes a page, prints the result, and stops the browser. To keep cookies and login state between runs, start a persistent profile instead; the scrape call is the same.

curl
Python

INTERNAL_UUID=$(curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/one_time" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" -d '{}' | jq -r .internal_uuid)

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "wait_for": "h1"}'

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

pip install requests

Save as scrape.py:

import os
import requests

base_url = os.environ["SURFSKY_API_BASE_URL"].rstrip("/")
headers = {"X-Cloud-Api-Token": os.environ["SURFSKY_API_TOKEN"]}

response = requests.post(f"{base_url}/profiles/one_time", headers=headers, json={}, timeout=120)
response.raise_for_status()
session = response.json()

result = requests.post(
    f"{base_url}/profiles/{session['internal_uuid']}/scrape",
    headers=headers,
    json={"url": "https://example.com", "wait_for": "h1"},
    timeout=130,
).json()
print(result["data"]["status"])
print(result["data"]["content"])

requests.post(f"{base_url}/profiles/{session['internal_uuid']}/stop", headers=headers, timeout=120)

python scrape.py

Expected output: 200 followed by HTML containing Example Domain.

Request parameters

POST /profiles/{internal_uuid}/scrape accepts:

Field	Default	Meaning
`url`	Required	Page to visit.
`wait_until`	`"domcontentloaded"`	Navigation condition: `commit`, `domcontentloaded`, `load`, or `networkidle`.
`wait_for`	Omitted	CSS or XPath selector to wait for after navigation.
`timeout`	`30000`	Navigation and selector timeout, in milliseconds.
`wait`	`0`	Extra delay after loading, in seconds, up to `60`.
`screenshot`	`false`	Include a Base64 PNG screenshot.
`auto_captcha_solve`	`false`	Solve CAPTCHAs during the scrape. Requires `anti_captcha` at browser start.
`human_actions`	`0`	Random browsing actions before returning, up to `3`.

wait_for with a selector for the content you need is more reliable than networkidle, which can time out on pages with background traffic. The whole request must finish within 120 seconds.

Read the response

{
  "success": true,
  "msg": "",
  "data": {
    "url": "https://example.com/",
    "status": 200,
    "status_text": "OK",
    "content": "<!doctype html><html>...</html>",
    "cookies": [],
    "screenshot": "iVBORw0KGgo..."
  }
}

success is about the Surfsky request. data.status is the target site’s HTTP status, so a 200 from Surfsky can carry a 403 or a challenge page from the site. Check both. screenshot is present only when requested; decode it with Base64 to get a PNG.

Scrape several pages

Send an array of request objects to the same endpoint:

[
  {"url": "https://example.com", "wait_for": "h1"},
  {"url": "https://example.org", "screenshot": true}
]

Pages run one after another in the same browser and share its cookies. data comes back as an array in request order; failed items carry error and status_code instead of a page. Keep a batch within the 120-second limit. For parallel work, use several browsers.

Stop the session

Stop the browser through the API with the internal_uuid from the start response, as the examples do:

curl --fail-with-body -X POST \
  "$SURFSKY_API_BASE_URL/profiles/INTERNAL_UUID/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

Otherwise the browser stops after the inactivity timeout, 30 seconds by default. Do not mix the Scraping API with CDP automation in the same browser; the scraper takes over the page and closes extra tabs.

Get started

Connect your tools

Browser sessions

Browser actions

CAPTCHA guides

Debugging

Performance and limits

About Surfsky

Prerequisites

Run an example

Request parameters

Read the response

Scrape several pages

Stop the session

​Prerequisites

​Run an example

​Request parameters

​Read the response

​Scrape several pages

​Stop the session

Prerequisites

Run an example

Request parameters

Read the response

Scrape several pages

Stop the session