Scraping API

POST /scrape loads one page in a browser that is already running and returns it as cleaned HTML, Markdown, links or a screenshot. The browser runs behind a residential proxy of the region you picked in the dashboard, with a desktop fingerprint, solves the challenge of a blocked page and keeps nothing between requests. Use the full flow instead when pages must share a login, when you need a specific country or city rather than the region, when you bring your own proxy or fingerprint, or when you need extensions or a persistent profile. To have a coding agent set this up in your project, give it the setup prompt.

Scrape a page

Set your API token and base URL. The endpoint is available on on-demand plans.

curl
Python (async)
JavaScript

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

Requires Python 3.12 or newer and surfsky 0.0.18 or newer.

import asyncio
from surfsky import AsyncSurfsky


async def main():
    async with AsyncSurfsky() as client:
        page = await client.scrape("https://example.com", formats=["markdown"])
        print(page.status, page.title)
        print(page.markdown)


asyncio.run(main())

Requires Node.js 24 or newer, or Bun, and surfsky 0.0.14 or newer.

import { Surfsky } from "surfsky";

const client = new Surfsky();
const page = await client.scrape("https://example.com", { formats: ["markdown"] });
console.log(page.status, page.title);
console.log(page.markdown);

A page that answers at once comes back in about a second. A page behind a challenge takes 10 to 60 seconds more. One request has a budget of 120 seconds in total.

Examples

# cleaned HTML, the default
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq -r .data.content

# the full rendered DOM: the cleaned HTML drops forms, scripts and everything outside the article
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://httpbin.org/forms/post", "formats": ["raw_html"]}' | jq -r .data.raw_html

# Markdown of the main content, without menus and footers
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["markdown"]}' | jq -r .data.markdown

# Markdown of the whole page: storefronts and home pages have no single main content
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["markdown"], "only_main_content": false}' | jq -r .data.markdown

# every link on the page, absolute
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["links"]}' | jq .data.links

# a full-page screenshot into a file
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["screenshot"]}' | jq -r .data.screenshot | base64 -d > apple.png

# a JPEG of the viewport only
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["screenshot"], "screenshot_options": {"full_page": false, "format": "jpeg", "quality": 70}}' | jq -r .data.screenshot | base64 -d > apple.jpg

# several formats at once; the fields outside the formats are always there
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["html", "markdown", "links"]}' | jq '.data | {final_url, status, title, content_type, metadata}'

# images, media and fonts are skipped by default; skip the stylesheets too for text only
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "formats": ["markdown"], "block_resources": ["image", "media", "font", "stylesheet"]}' | jq -r .data.markdown

# download everything, for a page that fills its content only once its images have loaded
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "block_resources": []}' | jq -r .data.content

# cookies set before navigation, and the jar after the page loaded
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://httpbin.org/cookies", "formats": ["html", "cookies"], "cookies": [{"domain": "httpbin.org", "name": "session", "value": "abc"}]}' | jq '.data | {content, cookies}'

# the jar of one request as the cookies of the next
JAR=$(curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://httpbin.org/cookies/set?session=abc", "formats": ["cookies"]}' | jq -c .data.cookies)

curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"url\": \"https://httpbin.org/cookies\", \"cookies\": $JAR}" | jq -r .data.content

# wait for the load event, then for an element, then one more second
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.apple.com", "wait_until": "load", "wait_for": "nav", "wait": 1, "timeout": 20000}' | jq .data.title

# a search typed into the page: click, type, a key, a wait for results, a scroll
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://duckduckgo.com/", "formats": ["links"], "actions": [
    {"type": "click", "selector": "textarea[name=q]"},
    {"type": "type", "selector": "textarea[name=q]", "text": "surfsky cloud browser"},
    {"type": "press", "key": "Enter"},
    {"type": "wait", "selector": "article"},
    {"type": "scroll", "direction": "down"},
    {"type": "wait", "ms": 500}
  ]}' | jq '.data | {final_url, title, links: .links[:10]}'

# a page behind a challenge: solved by default, 10 to 60 seconds longer
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://nopecha.com/demo/cloudflare", "timeout": 60000}' | jq '.data | {status, title}'

# the block page as the site sent it, no solve, no CAPTCHA charge
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://nopecha.com/demo/cloudflare", "auto_captcha_solve": false}' | jq '.data | {status, title}'

# a request that failed: HTTP 400, code names the cause, data carries the page the browser was left on
curl -s -X POST "$SURFSKY_API_BASE_URL/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "wait_for": "#never", "timeout": 5000}' | jq '{success, code, msg, page: .data.content[:80]}'

Request parameters

The same body works on POST /scrape and on POST /profiles/{internal_uuid}/scrape.

Field	Default	Meaning
`url`	Required	Page to visit.
`formats`	`["html"]`	What to return: `html` (cleaned), `raw_html` (the full rendered DOM), `markdown`, `links`, `screenshot`, `cookies`.
`only_main_content`	`true`	Strip navigation, footers and sidebars from `html` and `markdown`. Set `false` on storefronts and home pages, where the main content is not one article.
`screenshot_options`	`{"full_page": true, "format": "png"}`	`format` is `png` or `jpeg`; `quality` 1 to 100 applies to `jpeg`.
`block_resources`	Images, media and fonts	Resource types not downloaded: `image`, `media`, `font`, `stylesheet`. `[]` blocks nothing. Nothing is blocked when a screenshot is requested.
`cookies`	Omitted	Cookies set before navigation, in the JSON shape `cookies` returns or as Netscape text.
`wait_until`	`"domcontentloaded"`	Navigation condition: `commit`, `domcontentloaded`, `load`, or `networkidle`.
`wait_for`	Omitted	CSS or XPath selector to wait for after navigation.
`wait`	`0`	Extra delay after loading, in seconds, up to `60`.
`timeout`	`30000`	Navigation, selector and action timeout, in milliseconds.
`auto_captcha_solve`	`true`	Solve the challenge of a page that answers 403, 405, 429 or 503.
`actions`	`[]`	Up to 10 steps run on the loaded page with human emulation, in order.

wait_for with a selector for the content you need is more reliable than networkidle, which can time out on pages with background traffic. A solve adds 10 to 60 seconds within the 120-second budget. Each solve is charged at your plan’s CAPTCHA price, whether or not the page opens afterwards. With auto_captcha_solve: false a blocked page comes back as it is, with its status. A widget on a page that opened normally is never solved by this endpoint; use CAPTCHA solving in your own browser for those.

Actions

Action	Fields	What it does
`click`	`selector`	Moves the mouse to the element and clicks.
`type`	`selector`, `text`	Clicks the element and types the text key by key.
`press`	`key`	Presses one key, such as `Enter` or `Tab`.
`scroll`	`selector` or `direction`	Scrolls the element into view, or one screen `up` or `down`.
`wait`	`selector` or `ms`	Waits for an element to appear, or for a pause up to 60000 ms.

The first failing action ends the request with action_failed; the response data carries the page the browser was left on. A block page that a click leads to is solved like the first one.

Read the response

{
  "success": true,
  "msg": "",
  "data": {
    "url": "https://example.com",
    "final_url": "https://example.com/",
    "status": 200,
    "status_text": "OK",
    "title": "Example Domain",
    "content_type": "text/html",
    "metadata": {
      "description": null,
      "language": "en",
      "canonical_url": null,
      "og_title": null,
      "og_description": null,
      "og_image": null
    },
    "content": "<div><p>This domain is for use in documentation examples...</p></div>"
  }
}

data.status is the target site’s HTTP status after redirects and solves, so a 200 from Surfsky can carry a 403 from the site. Each requested format adds its own key: content for html, raw_html, markdown, links (absolute URLs, deduplicated), screenshot (Base64) and cookies (the jar after the scrape, accepted as cookies of the next request). A response to an error status with an empty body has that status and empty content. A request that could not finish answers HTTP 400 with success: false and a code:

Code	Meaning
`page_load_timeout`	The page did not reach `wait_until` within `timeout`.
`selector_timeout`	`wait_for` did not appear within `timeout`.
`site_unreachable`	DNS, TLS or a reset connection; `data.error` carries the browser’s reason.
`action_failed`	One action timed out or found no element; `msg` says which.
`captcha_solve_failed`, `captcha_solve_timeout`	The challenge was not passed; `data` carries the page.
`scrape_timeout`	The 120-second budget ran out.

on_demand_plan_required (403) means the token’s plan is not on-demand, insufficient_balance (402) an empty balance, parallel_browsers_limit_reached (429) that every browser slot of the account is in use. A request in flight holds a slot like a running browser.

Billing

A request is one session, billed in 30-second steps of browser time. For example, a 20-second scrape costs 30 seconds and a 70-second one costs 90. Proxy traffic is charged per GB at your plan’s premium rate, and each CAPTCHA solve at its price. A page the site refused, an unreachable site and a browser that failed on our side are not charged.

When to run your own browser

Start a browser, scrape through it and stop it. POST /profiles/{internal_uuid}/scrape takes the body above, and also an array of bodies: the pages load one after another in one browser, and data comes back as an array in that order, with error, code and status_code in place of a failed page. The 120 seconds cover the whole batch.

INTERNAL_UUID=$(curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/one_time" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" -d '{}' | jq -r .internal_uuid)

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/scrape" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://example.com", "wait_for": "h1"},
       {"url": "https://example.org", "formats": ["screenshot"]}]'

curl -s -X POST "$SURFSKY_API_BASE_URL/profiles/$INTERNAL_UUID/stop" \
  -H "X-Cloud-Api-Token: $SURFSKY_API_TOKEN"

In the SDKs this is client.profiles.scrape(session, url, ...) inside client.session(), which stops the browser when the scope exits. Stop the browser yourself after direct HTTP requests, or it stops after the inactivity timeout. Do not mix scrape calls with CDP automation in the same browser: the scraper takes over the page and closes extra tabs. For input sequences longer than ten actions, drive the browser through the SDK’s human emulation methods instead.

Get started

Connect your tools

Browser sessions

Browser actions

CAPTCHA guides

Debugging

Performance and limits

About Surfsky

Scrape a page

Examples

Request parameters

Actions

Read the response

Billing

When to run your own browser

​Scrape a page

​Examples

​Request parameters

​Actions

​Read the response

​Billing

​When to run your own browser

Scrape a page

Examples

Request parameters

Actions

Read the response

Billing

When to run your own browser