Scraping API

POST /scrape opens a page in a ready browser and returns it as cleaned HTML, Markdown, links or a screenshot. The browser uses a residential proxy in the region set in your dashboard and a desktop fingerprint. It passes challenge pages and solves captchas on its own, and keeps nothing between requests. Run your own browser instead if pages need to share a login, or you need a specific country or city, your own proxy or fingerprint, extensions or a persistent profile. A coding agent can set this up from the setup prompt.

Scrape a page

Set your API token and base URL first.

curl
Python (async)
JavaScript

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

Requires Python 3.12 or newer and surfsky 0.0.18 or newer.

import asyncio
from surfsky import AsyncSurfsky


async def main():
    async with AsyncSurfsky() as client:
        page = await client.scrape("https://example.com", formats=["markdown"])
        print(page.status, page.title)
        print(page.markdown)


asyncio.run(main())

Requires Node.js 24 or newer, or Bun, and surfsky 0.0.14 or newer.

import { Surfsky } from "surfsky";

const client = new Surfsky();
const page = await client.scrape("https://example.com", { formats: ["markdown"] });
console.log(page.status, page.title);
console.log(page.markdown);

Most pages come back in a second or two, a challenge page adds 10 to 60 seconds. A request gets 120 seconds in total.

Recipes

Each recipe is a single request. The SDK examples reuse the client from Scrape a page: body fields go in as keyword arguments in Python or an options object in JavaScript, and the result has the fields of data from Read the response.

An article as Markdown, for an LLM

Markdown of the main content, without menus, footers and sidebars. Headings and links stay.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://en.wikipedia.org/wiki/Web_scraping", "formats": ["markdown"]}' | jq -r .data.markdown

page = await client.scrape("https://en.wikipedia.org/wiki/Web_scraping", formats=["markdown"])
print(page.title, len(page.markdown))

const page = await client.scrape("https://en.wikipedia.org/wiki/Web_scraping", { formats: ["markdown"] });
console.log(page.title, page.markdown.length);

A product page: name and price

wait_for waits for the price. The request returns as soon as it renders, or fails with selector_timeout if it never does.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "formats": ["markdown"], "wait_for": ".price_color"}' | jq -r '.data.title, .data.markdown'

import re

page = await client.scrape(
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    formats=["markdown"],
    wait_for=".price_color",
)
print(page.title, re.search(r"£\d+\.\d\d", page.markdown).group())

const page = await client.scrape("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", {
  formats: ["markdown"],
  wait_for: ".price_color",
});
console.log(page.title, page.markdown.match(/£\d+\.\d\d/)[0]);

The links of a front page

links are absolute and deduplicated. This example keeps only links to other sites.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com", "formats": ["links"]}' | jq '.data.links | map(select(contains("ycombinator.com") | not))'

page = await client.scrape("https://news.ycombinator.com", formats=["links"])
print([link for link in page.links if "ycombinator.com" not in link])

const page = await client.scrape("https://news.ycombinator.com", { formats: ["links"] });
console.log(page.links.filter((link) => !link.includes("ycombinator.com")));

A screenshot of a page

A screenshot is taken only when formats includes screenshot; data.screenshot then holds a Base64 PNG of the whole page. It also loads images, fonts and media, so the request usually costs about twice as much. Pass block_resources to skip what the shot can do without.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["screenshot"]}' | jq -r .data.screenshot | base64 -d > apple.png

import base64

page = await client.scrape("https://www.apple.com", formats=["screenshot"])
open("apple.png", "wb").write(base64.b64decode(page.screenshot))

import { writeFile } from "node:fs/promises";

const page = await client.scrape("https://www.apple.com", { formats: ["screenshot"] });
await writeFile("apple.png", Buffer.from(page.screenshot, "base64"));

A search typed into a page

Actions run on the loaded page with human-like input: click the search box, type the query, press Enter, wait for results, scroll. You get the page as the actions left it.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://duckduckgo.com/", "formats": ["links"], "actions": [
    {"type": "click", "selector": "textarea[name=q]"},
    {"type": "type", "selector": "textarea[name=q]", "text": "surfsky cloud browser"},
    {"type": "press", "key": "Enter"},
    {"type": "wait", "selector": "article"},
    {"type": "scroll", "direction": "down"},
    {"type": "wait", "ms": 500}
  ]}' | jq '.data | {final_url, title, links: .links[:10]}'

page = await client.scrape(
    "https://duckduckgo.com/",
    formats=["links"],
    actions=[
        {"type": "click", "selector": "textarea[name=q]"},
        {"type": "type", "selector": "textarea[name=q]", "text": "surfsky cloud browser"},
        {"type": "press", "key": "Enter"},
        {"type": "wait", "selector": "article"},
        {"type": "scroll", "direction": "down"},
        {"type": "wait", "ms": 500},
    ],
)
print(page.final_url, page.links[:10])

const page = await client.scrape("https://duckduckgo.com/", {
  formats: ["links"],
  actions: [
    { type: "click", selector: "textarea[name=q]" },
    { type: "type", selector: "textarea[name=q]", text: "surfsky cloud browser" },
    { type: "press", key: "Enter" },
    { type: "wait", selector: "article" },
    { type: "scroll", direction: "down" },
    { type: "wait", ms: 500 },
  ],
});
console.log(page.final_url, page.links.slice(0, 10));

A listing that loads on scroll

Wait after each scroll so the next batch can load. The result has everything loaded by the end.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/scroll", "actions": [
    {"type": "scroll", "direction": "down"}, {"type": "wait", "ms": 1500},
    {"type": "scroll", "direction": "down"}, {"type": "wait", "ms": 1500},
    {"type": "scroll", "direction": "down"}, {"type": "wait", "ms": 1500}
  ]}' | jq -r .data.content | grep -c 'class="quote"'

scroll = [{"type": "scroll", "direction": "down"}, {"type": "wait", "ms": 1500}]
page = await client.scrape("https://quotes.toscrape.com/scroll", actions=scroll * 3)
print(page.content.count('class="quote"'))

const scroll = [{ type: "scroll", direction: "down" }, { type: "wait", ms: 1500 }];
const page = await client.scrape("https://quotes.toscrape.com/scroll", { actions: [...scroll, ...scroll, ...scroll] });
console.log(page.content.split('class="quote"').length - 1);

A page behind Cloudflare

The challenge page is passed before the page is captured, with a captcha solve if it has one. challenge in the response says what was there and what it cost. Give the request a longer timeout for the wait. A challenge that passes by itself is free.

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://nopecha.com/demo/cloudflare", "timeout": 60000}' | jq '.data | {status, title, challenge}'

page = await client.scrape("https://nopecha.com/demo/cloudflare", timeout=60000)
print(page.status, page.title, page.challenge)

const page = await client.scrape("https://nopecha.com/demo/cloudflare", { timeout: 60000 });
console.log(page.status, page.title, page.challenge);

POST /scrape keeps nothing between requests, so for pages behind a login start your own browser. POST /profiles/{internal_uuid}/scrape takes the same body. Here the first request logs in, the second reads a page as that user. The demo site accepts any name and password.

INTERNAL_UUID=$(curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/one_time" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" -d '{}' | jq -r .internal_uuid)

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/login", "actions": [
        {"type": "type", "selector": "#username", "text": "reader"},
        {"type": "type", "selector": "#password", "text": "secret"},
        {"type": "click", "selector": "input[type=submit]"},
        {"type": "wait", "selector": "a[href=\"/logout\"]"}
      ]}' > /dev/null

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/", "formats": ["markdown"]}' | jq '.data.markdown | contains("Logout")'

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

async with client.session() as session:
    await client.profiles.scrape(
        session,
        "https://quotes.toscrape.com/login",
        actions=[
            {"type": "type", "selector": "#username", "text": "reader"},
            {"type": "type", "selector": "#password", "text": "secret"},
            {"type": "click", "selector": "input[type=submit]"},
            {"type": "wait", "selector": "a[href='/logout']"},
        ],
    )
    page = await client.profiles.scrape(session, "https://quotes.toscrape.com/", formats=["markdown"])
    print("Logout" in page.markdown)

await using session = await client.session();
await client.profiles.scrape(session, "https://quotes.toscrape.com/login", {
  actions: [
    { type: "type", selector: "#username", text: "reader" },
    { type: "type", selector: "#password", text: "secret" },
    { type: "click", selector: "input[type=submit]" },
    { type: "wait", selector: "a[href='/logout']" },
  ],
});
const page = await client.profiles.scrape(session, "https://quotes.toscrape.com/", { formats: ["markdown"] });
console.log(page.markdown.includes("Logout"));

Request parameters

The same body works on POST /scrape and on POST /profiles/{internal_uuid}/scrape.

Field	Default	Meaning
`url`	Required	Page to visit.
`formats`	`["html"]`	What to return: `html` (cleaned), `raw_html` (the full rendered DOM), `markdown`, `links`, `screenshot`, `cookies`.
`only_main_content`	`true`	Remove headers, navigation, footers, sidebars, menus, breadcrumbs, cookie banners, modals, ads and share widgets from `html` and `markdown`, found by tag, class or id. A block that contains `#main` stays. Short pages and listings keep their content.
`screenshot_options`	`{"full_page": true, "format": "png"}`	`full_page` takes the whole height of the page at the window’s width. `format` is `png` or `jpeg`; `quality` 1 to 100 applies to `jpeg`.
`block_resources`	Images, media, fonts, frames from other sites and ads	What not to download: `image`, `media` (including streamed audio and video), `font`, `stylesheet`, `iframe` (frames from other sites) and `ads` (ads, analytics, cookie banners). Captcha and anti-bot frames always load. `[]` blocks nothing. With a screenshot the default blocks nothing; `["ads"]` gives a screenshot without ads.
`cookies`	Omitted	Cookies set before navigation, in the JSON shape `cookies` returns or as Netscape text.
`wait_until`	`"domcontentloaded"`	When navigation counts as done: `commit`, `domcontentloaded`, `load` or `networkidle`.
`wait_for`	Omitted	CSS or XPath selector to wait for after navigation. It passes when any element matching it is visible.
`wait`	`0`	Extra delay after loading, in seconds, up to `60`.
`timeout`	`30000`	Navigation, selector and action timeout, in milliseconds.
`auto_captcha_solve`	`true`	Pass challenge pages and solve captchas, paid types included. A solve adds 10 to 60 seconds and is charged whatever the outcome. `false` solves nothing, `challenge` still says what was there. See Challenge pages and captchas.
`actions`	`[]`	Up to 10 steps run in order on the loaded page, with human-like input.

Examples per option

# the full rendered DOM, with scripts and styles
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://httpbin.org/forms/post", "formats": ["raw_html"]}' | jq -r .data.raw_html

# a JPEG of the viewport only; the default is a PNG of the full page
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["screenshot"], "screenshot_options": {"full_page": false, "format": "jpeg", "quality": 70}}' | jq -r .data.screenshot | base64 -d > apple.jpg

# several formats in one request; status, title and metadata always come back
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["html", "markdown", "links"]}' | jq '.data | {final_url, status, title, content_type, metadata}'

# text only: skip stylesheets too (images, media, fonts, iframes and ads are skipped by default)
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["markdown"], "block_resources": ["image", "media", "font", "iframe", "ads", "stylesheet"]}' | jq -r .data.markdown

# download everything, for pages that show content only after images load
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "block_resources": []}' | jq -r .data.content

# cookies set before navigation, and the jar after the page loaded
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://httpbin.org/cookies", "formats": ["html", "cookies"], "cookies": [{"domain": "httpbin.org", "name": "session", "value": "abc"}]}' | jq '.data | {content, cookies}'

# pass the cookies of one request to the next
JAR=$(curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://httpbin.org/cookies/set?session=abc", "formats": ["cookies"]}' | jq -c .data.cookies)

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"url\": \"https://httpbin.org/cookies\", \"cookies\": $JAR}" | jq -r .data.content

# wait for the load event, then for an element, then one more second
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "wait_until": "load", "wait_for": "nav", "wait": 1, "timeout": 20000}' | jq .data.title

# the challenge page as is: nothing solved or charged, challenge says what it is
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://nopecha.com/demo/cloudflare", "auto_captcha_solve": false}' | jq '.data | {status, challenge}'

# a failed request: HTTP 400, code says why, data has the page as the browser left it
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "wait_for": "#never", "timeout": 5000}' | jq '{success, code, msg, page: .data.content[:80]}'

Which wait to use

wait_until sets when navigation is done. The default domcontentloaded returns once the HTML is parsed, before late images and scripts, so a page drawn by scripts can still be blank in a screenshot. load waits for those too. networkidle waits for the network to go quiet and times out on pages with constant background traffic like analytics or chat widgets. commit returns on the first response, enough for a status check.
wait_for waits for one element. Try it first: point it at the content you need, such as a listing, a price or an article, and the request returns as soon as it appears.
wait is a fixed pause. Use it only when nothing on the page signals it is ready, since it adds time to every request.
actions are for pages that show content only after input: a search, a cookie prompt, a “load more” button, a tab. A wait action between steps waits for the result of the previous one.

A request runs in this order:

Load url until wait_until, within timeout.
Pause for wait seconds, then wait for wait_for, within timeout.
Run actions in order.
Capture formats.

After navigation and after every step the page is checked: a challenge page is passed and a captcha is solved. If a wait ended with a solve, it runs again. If a wait_for element doesn’t show up, the page is checked, and after a solve the request waits for the element once more. Actions are never interrupted or repeated: a captcha that shows up during an action is handled by the check after it.

Actions

Action	Fields	What it does
`click`	`selector`	Moves the mouse to the element and clicks.
`type`	`selector`, `text`	Clicks the element and types the text key by key.
`press`	`key`	Presses one key, such as `Enter` or `Tab`.
`scroll`	`selector` or `direction`	Scrolls the element into view, or one screen `up` or `down`.
`wait`	`selector` or `ms`	Waits for an element to become visible, or for a pause up to 60000 ms.
`solve`		Solves the captcha on the page at that point, even with `auto_captcha_solve: false`.

If an action fails, the request ends with action_failed, and data has the page as the browser left it, plus challenge. If a click leads to a challenge page, the check after the click passes it.

Challenge pages and captchas

Three things can get in the way. A challenge page is what an anti-bot service shows instead of the page until the browser passes its check, usually with status 403, 405, 429 or 503. Some pass by themselves, some have a captcha. A block page is a refusal no check gets past, for example a banned exit IP, and is reported right away. A page captcha sits on the page itself and guards a form or an action. With auto_captcha_solve: true (the default) every challenge page and captcha the checks find is solved, paid types included. An invisible captcha runs only when the site calls it, usually when a form is sent. Until then there is nothing to solve and the request doesn’t wait for it; challenge shows it with solved: false if the check sees it. After a click that sends the form, the check after the click solves it. A captcha that appears after the last step isn’t seen, so end with a wait, wait_for or solve step. For a form behind a captcha you can also drive the browser over CDP and call Captcha.solve yourself. Every solve is a captcha charge, whatever the outcome. A challenge page that passes by itself is free.

If your content isn’t behind a challenge page, auto_captcha_solve: false is faster: captchas on the page are left alone and not charged.

If a challenge page can’t be passed, or a block page comes up, POST /scrape retries once in another browser with a different exit IP. A connection error (reset, closed or timed out) is retried the same way; a certificate error is not. A domain that doesn’t exist looks like a reset connection from the browser, so it is retried too, and comes back as site_unreachable after the second attempt. If it’s still blocked, you get the page as is with the site’s status, and the request isn’t billed for browser time or traffic. An unsolved page captcha doesn’t hold the request: the page comes back right away with challenge.solved: false. With auto_captcha_solve: false nothing is solved or waited for. You get the page as is, challenge still says what was there, and a challenge or block page returned this way isn’t billed for browser time. To choose the moment yourself, keep false and add a solve action where you expect the captcha. It solves what’s on the page at that point, including a widget still loading, and does nothing if there’s nothing to solve. A challenge page the action can’t pass comes back as is, without a retry.

Read the response

{
  "success": true,
  "msg": "",
  "data": {
    "url": "https://example.com/",
    "final_url": "https://example.com/",
    "status": 200,
    "status_text": "OK",
    "title": "Example Domain",
    "content_type": "text/html",
    "challenge": null,
    "metadata": {
      "description": null,
      "language": "en",
      "canonical_url": null,
      "og_title": null,
      "og_description": null,
      "og_image": null
    },
    "content": "<html lang=\"en\"><head><title>Example Domain</title></head><body><div><h1>Example Domain</h1><p>This domain is for use in documentation examples...</p></div></body></html>",
    "receipt": {
      "session_uuid": "5f0c1d7e9a3b4c62b1d08e4f7a6c2d19",
      "billed_seconds": 10,
      "billed_minutes": 0.17,
      "traffic_bytes": 18342,
      "browser_cost": "0.000972",
      "traffic_cost": "0.000183",
      "captcha_cost": "0",
      "total_cost": "0.001155"
    }
  }
}

data.status is the site’s HTTP status after redirects and solves, so a 200 from Surfsky can carry a 403 from the site. Each format you ask for adds a key: content for html, plus raw_html, markdown, links (absolute, deduplicated), screenshot (Base64) and cookies (pass them as cookies to the next request). Links and image addresses in content and markdown are absolute; raw_html is the document as the browser has it. If the site answered with an error and an empty body, you get that status and empty content. A PDF comes back with its content_type and no text: its text isn’t extracted. challenge is null if nothing got in the way, otherwise it describes the last obstacle:

{
  "kind": "challenge_page",
  "captcha": "turnstile",
  "solved": true,
  "cost": "0"
}

kind is challenge_page, block_page or page_captcha. captcha is the captcha type as named in Captcha.solve, or null for a challenge page without one. solved is true if the request got past it, by a solve or on its own, and false if it was still there at the end. cost is the total captcha charge for the request as a string, "0" if nothing was charged. receipt is what the request cost: the browser time billed, the traffic of the page, the captcha solves, and their total in USD. session_uuid is the request’s entry in session receipts and in the ledger. A request that isn’t billed has zeros for time and traffic. After a retry the receipt is of the final attempt, with the captcha solves of both. total_cost here includes traffic and captchas; in session receipts it is browser time only, and traffic and captchas are ledger entries of their own. It is on error responses too, and missing only when the request never reached a browser. A request that couldn’t finish returns HTTP 400 with success: false and a code:

Code	Meaning	Charged
`page_load_timeout`	The page did not reach `wait_until` within `timeout`.	Yes, if the page came back with content. A blank page is free. If the 120-second budget cut the load short, it is free and no page comes back.
`selector_timeout`	No element matching `wait_for` became visible within `timeout`.	Yes, the page is in `data`.
`site_unreachable`	DNS, TLS or connection error; `data.error` has the browser’s reason, `net::ERR_CONNECTION_RESET` for a domain that doesn’t exist.	No.
`action_failed`	One action timed out or found no element; `msg` says which.	Yes, the page is in `data`.
`scrape_timeout`	The 120-second budget ran out.	No.

Other errors:

Code	Status	Meaning
`insufficient_balance`	402	The balance is empty; GET /billing/balance shows what is left.
`parallel_browsers_limit_reached`	429	All browser slots are busy. A request in progress takes a slot like a running browser.
`rate_limits_reached`	429	The minute budget, 60 requests per browser slot, is used up.

In the SDKs a failed request raises APIError with the same status_code and code. body has the full response, page included:

from surfsky import APIError

try:
    await client.scrape("https://example.com", wait_for="#never", timeout=5000)
except APIError as error:
    print(error.status_code, error.code, error.body["data"]["content"][:80])

import { APIError } from "surfsky";

try {
  await client.scrape("https://example.com", { wait_for: "#never", timeout: 5000 });
} catch (error) {
  if (error instanceof APIError) console.log(error.statusCode, error.code, error.body.data.content.slice(0, 80));
}

Billing

You pay for a request when its response carries the page. That includes an error of your own wait_for, wait or action on a loaded page, and a load your own timeout cut short. A response with no page is free: a page the site refused, an unreachable site, a blank page, and a request we could not finish in time. A CAPTCHA solve is charged whatever became of the page. Browser time is billed in 10-second steps at your plan’s browser-hour rate, from the start of the page’s work to its delivery: a 4-second scrape counts as 10 seconds, a 25-second one as 30. Traffic is the page’s own, billed per GB, and CAPTCHA solves are billed at their own price. The price is the same whichever browser serves the request. On a reserved plan the monthly fee covers the browser time; traffic and solves still come from the balance. The response’s receipt has the amounts, and the charges are in the ledger. Billing covers the plans and where to see usage. Per plan, at the rates on the pricing page. A light page is 200 KB in one step, a typical page is 12 seconds and 0.9 MB.

Plan	Browser slots	Requests a minute at most	Pages per $1, light	Pages per $1, typical	Monthly credit, typical
PAYG	10	600	340	100	-
Starter	15	900	380	110	3,200
Base	50	3,000	440	125	12,300
Advanced	199	11,940	510	145	28,500

A request over your browser slots gets 429 parallel_browsers_limit_reached.

When to run your own browser

Start a browser, scrape through it, then stop it. POST /profiles/{internal_uuid}/scrape takes the same body, one page per request, and the pages share the browser’s cookies. The login recipe shows the calls in curl and the SDKs. In the SDKs client.session() stops the browser when the block ends. With plain HTTP, stop it yourself, or it stops after the inactivity timeout. Don’t mix scrape calls and CDP automation in one browser: the scraper takes over the first tab. For more than ten input steps, use the SDK’s human emulation methods.

Get started

Connect your tools

Browser sessions

Browser actions

CAPTCHA guides

Debugging

Performance and limits

About Surfsky

Scrape a page

Recipes

An article as Markdown, for an LLM

A product page: name and price

The links of a front page

A screenshot of a page

A search typed into a page

A listing that loads on scroll

A page behind Cloudflare

Request parameters

Examples per option

Which wait to use

Actions

Challenge pages and captchas

Read the response

Billing

When to run your own browser

Scrape a page

Recipes

An article as Markdown, for an LLM

A product page: name and price

The links of a front page

A screenshot of a page

A search typed into a page

A listing that loads on scroll

A page behind Cloudflare

Pages behind a login, in one browser

Request parameters

Examples per option

Which wait to use

Actions

Challenge pages and captchas

Read the response

Billing

When to run your own browser