medium severityScrapingBee API (fetch_url or similar web scraping tools using ScrapingBee)
API request fails with timeout error after 30-40 seconds when using `render_js=true`, `js_scenario`, or `wait` parameters on JavaScript-heavy sites. Response may return incomplete HTML missing dynamic content, or error like "timeout exceeded" in SDK. Builder sees: failed scrape, partial/missing data from JS-rendered elements.
Root cause
JavaScript-heavy pages require time to render dynamic content, but ScrapingBee has strict limits: base API timeout ~30s (with retries), `wait` max 35s, `js_scenario` max 40s execution. Default 2s wait is insufficient; network activity or missing elements cause premature returns or timeouts. Client SDK timeouts may also trigger if not configured higher.
ScrapingBeetimeoutJavaScriptrender_jswaitjs_scenarioheadless browser