Skip to main content

Troubleshooting

In this section, we'll cover common issues (blocking, CAPTCHA, rate limiting, slow performance, etc.) and how to troubleshoot them.

Best Practices​

  • Use DevTools for debugging page interactions
  • Monitor network requests and console logs
  • Test scripts with smaller concurrency settings first:
    • Start with single instance runs
    • Use lower concurrency settings
    • Gradually increase scale after confirming stability
  • Try different proxy types:
    • Server proxies may suffice for basic websites
    • Quality residential proxies needed for heavy anti-bot systems
    • See our Proxy Configuration Guide for detailed recommendations
  • If encountering frequent CAPTCHAs:
    • Check if the website shows CAPTCHAs in a regular Chrome browser (some sites, like those under Cloudflare's Under Attack Mode, show CAPTCHAs to all visitors)
    • See our CAPTCHA Solving Guide for optimization tips

Handling Website Blocks​

If you encounter blocking issues (specific 4xx/5xx errors, blocking pages, or frequent CAPTCHAs), consider these key factors:

  1. Proxy Quality and Type

    • Data Center (DC) proxies are easily detected by sophisticated anti-bot systems
    • High-quality residential proxies provide better success rates as they represent real user IPs
    • Check IP quality scores using services like IPQualityScore to assess proxy reliability and risk level
    • Match proxy location to website's target market (essential for region-restricted platforms like Shopee)
    • Use proxies from appropriate cities/regions for location-sensitive services

  2. Request Frequency

    • Implement natural delays between actions
    • Avoid rapid sequential requests to the same endpoints
    • Consider using random intervals to mimic human behavior
    • Monitor and respect:
      • Website's internal rate limits
      • Anti-bot system rate thresholds
      • API endpoint-specific limitations

  3. Browser Profile Configuration

    • Try accessing the site with standard latest Chrome using the same proxy to compare behavior
    • If using custom fingerprints, try different configurations
    • Ensure your fingerprint parameters remain consistent throughout the session
    • Consider enabling specific browser features:
      • Local Storage and IndexedDB: Required for sites that store login sessions and user data (profiles will log out between sessions if disabled)
      • Service Workers: Needed for sites that cache pages or store auth data (e.g., Telegram). Note: enabling increases profile size significantly

  4. User Behavior Patterns

    • Avoid direct page access for sensitive targets. Try to find happy path for the website: Spoof referrer, follow natural website flow (use native navigation)
    • Implement natural user actions:
      • Random cursor movements (use bezier curves for smooth, human-like trajectories)
      • Natural scrolling patterns (varying speed, occasional stops, smooth acceleration/deceleration)
      • Realistic timing between actions
      • Human-like navigation paths through the site
    • Study and replicate real user behavior:
      • Record and analyze real user sessions
      • Create behavior graphs showing typical user flows
      • Model your automation after these authentic patterns
      • Track common paths and interaction frequencies

  5. Handling CAPTCHAs and Blocks If you encounter frequent CAPTCHAs or blocks:

    • Try creating new profiles to get fresh fingerprints
    • Adjust fingerprint settings if needed See API Reference
Support Information

If you're experiencing frequent or persistent errors, please provide the following information to our support team for a thorough investigation and resolution:

  • Request headers and body
  • Error response headers and body
  • x-cloud-tracing-uuid header value
Fingerprint Modification

Browser profile settings are generated with optimal compatibility in mind. The default settings are suitable for most use cases. However, if you need to modify fingerprints:

  • Default fingerprint parameters are optimized for maximum compatibility
  • Changing certain parameters (like user-agent) may compromise profile integrity
  • Only modify settings if you fully understand the implications

Slow Performance​

When debugging performance issues, it's important to understand the impact of network latency:

  1. Local vs Cloud Differences. Local testing often shows faster performance due to near-zero latency, while cloud execution includes network Round Trip Time (RTT). Each browser command requires a network round trip, and seemingly simple operations may require multiple CDP commands behind the scenes. For example, a simple click operation might involve 3-4 CDP commands (get element, scroll into view, click), each with its own RTT.

    Understanding the timing differences is crucial: local execution typically takes 1-5ms per command, while cloud execution averages 50-100ms RTT per command. Adding proxies introduces an additional 100-300ms latency per request. Consider a typical login flow with 10 commands: locally it might complete in 50ms (10 commands at 5ms each), but in the cloud it would take at least 1000ms (10 commands at 100ms each), and with proxies this could extend to 3000ms or more (10 commands at 300ms each including proxy latency).

  2. Browser Session Management. Browser session configuration significantly impacts performance. While multi-tab sessions are supported, they often lead to degraded performance - instead, use multiple single-tab sessions for better results. Whenever possible, reuse existing sessions rather than creating new ones to reduce overhead. You can also improve performance by disabling unnecessary features: consider blocking unwanted assets (CSS, images, JavaScript) and configuring proxies only when required. Using the waitUntil: "domcontentloaded" option can also speed up page loading significantly.

  3. Optimization Strategies. Parallel processing is crucial for optimal performance. Rather than executing tasks sequentially, implement concurrent processing to achieve significant performance gains. Authentication flows can be optimized by saving and reusing cookies to bypass repeated login processes. When choosing automation frameworks, consider WebSocket-based options like Playwright or Puppeteer over HTTP-based alternatives, as they typically offer better performance characteristics. Each framework also offers specific optimizations - for example, using locator.fill() instead of type() in Playwright for faster input operations.

    For tasks requiring persistent state or authentication, consider using persistent profiles to maintain cookies and session data between runs. This approach can significantly reduce overhead by:

    • Eliminating repeated login processes
    • Maintaining authenticated states
    • Preserving cookies and local storage

  4. Infrastructure Considerations. The physical distance between your code and Surfsky infrastructure plays a crucial role in performance. Running sessions in your geographic region can significantly reduce latency. Remember that additional services like CAPTCHA solving and proxy usage will add extra latency to your operations. The choice of runtime environment can also impact performance, with different languages and runtimes offering varying benefits for different types of workloads. For optimal latency between your infrastructure and Surfsky, please contact our support team to discuss deployment options near our clusters for low-latency connections.

Performance Optimization

For detailed information about optimizing performance and managing latency, including real-world performance comparisons and optimization strategies, see our Speed Optimization and Parallelization Guide.

Chrome DevTools Protocol (CDP) Errors​

When working with browser automation, you may encounter CDP-related errors in logs. The most common ones are:

  1. Protocol error Target.detachFromTarget: Target closed

    • Occurs when CDP tries to detach from a target (page/frame) that's already closed
    • This is normal during page navigation or tab closure
    • It's a race condition where the target closes before the detach command arrives

  2. Protocol error (Target.sendMessageToTarget): No session with given id

    • Occurs when trying to send a message to a CDP session that no longer exists
    • Common during browser/page closure
    • Another race condition where the session is closed but messages are still being sent

These errors are part of normal CDP operation and don't need to be "fixed" on Surfsky's server side because:

  • They are part of CDP's normal functionality
  • They don't affect automation functionality
  • They occur due to CDP and WebSocket's asynchronous nature

You can safely ignore these errors in your logs as they don't indicate actual problems.