I tested six web scraping APIs, from simple request-in, HTML-out services to scraping platforms and an AI-native API. Here is what each one does well, where it falls short, and which one I would choose for different projects.
Last reviewed: 18th October 2026. The prices, the credit rates and all the other parameters in this category are in such a state of flux that you simply have to check each provider's current documentation before you make a decision.
TL;DR
- Simplest "URL in, data out" call with JS rendering and AI extraction: ScrapingBee.
- Heavily protected sites at enterprise scale: Bright Data.
- LLM-ready Markdown and JSON for RAG and agents: Firecrawl.
- Ready-made scrapers and custom scraping programs on one platform: Apify.
- Proxy-backed scraping with target templates: Decodo.
- General-purpose scraping with JS rendering and residential proxies: WebScrapingAPI.
Running your own scraping stack involves managing headless browsers, proxies (rotating), CAPTCHA solving, retries, and the constant battle against anti-bot systems. A web scraping API offloads all that work to someone else. You just provide them with a URL, and you get the page back.
But these six services aren't interchangeable. Some return raw HTML and leave parsing to you. Others return structured JSON or LLM-ready Markdown. One runs whole scraping programs on its own platform. Their pricing models also behave very differently as volume grows.
I ran a comparison of 6 services across the three groups: request-in, HTML-out APIs, scraping platforms, and the AI-native API. It's an editorial comparison based on the services' documented capabilities and publicly available product info, not a controlled benchmark in which each API would have been scraping the same pages.
My big take away is that the best scraping API is the one whose pricing, output format and failure mode still make sense at ten times your initial volume.
What is a web scraping API?
A web scraping API is an HTTP endpoint which will fetch you a web page. You give it a URL to scrape and a few options and it will then route the request via a rotating proxy, render any JavaScript that's required in a headless browser, deal with retries and blocks and return the result in HTML, JSON or Markdown.
For example, one call can return the fully rendered HTML of a product page from a particular country, without you having to run a browser or buy a single proxy.
You trade control for convenience. The various services offer different things in return: raw HTML that you have to parse yourself, data that's been extracted via extraction rules or pre-built parsers, or clean Markdown that you can feed into an LLM. That's the thing that should determine which service you pick.
How a scraping API request works
Every service in this guide runs roughly the same pipeline behind its endpoint:
| Stage | What happens | What you control |
|---|---|---|
| 1. Request | You send the target URL plus parameters | Rendering, geography, device, wait conditions |
| 2. Proxy selection | The service picks an IP from a datacenter, residential, or mobile pool | Proxy tier, country, session reuse |
| 3. Fetch and render | A plain HTTP fetch, or a headless browser that runs JavaScript | Whether to render, what to wait for |
| 4. Anti-bot handling | Retries, fingerprinting, and CAPTCHA solving | Usually only a tier switch |
| 5. Response | HTML, structured JSON, or Markdown | Extraction rules, formats |
The request shape is also similar across vendors, which makes switching easier than it looks. Here is how the same ideas appear in each tool:
| Tool | Request style | Auth | JS rendering | Proxy tiers | Structured output |
|---|---|---|---|---|---|
| ScrapingBee | GET to the HTML API | Bearer header or api_key
|
render_js (on by default) |
premium_proxy, stealth_proxy
|
extract_rules, ai_query
|
| Decodo | POST to the v2 scrape endpoint | Basic auth token |
headless set to html
|
proxy_pool standard or premium |
parse, target templates |
| WebScrapingAPI | GET to the v2 endpoint |
api_key parameter |
render_js |
proxy_type datacenter or residential |
extract_rules |
| Bright Data | POST to the request endpoint with a zone | Bearer header | Handled by the zone, or Browser API | Chosen by zone type | Scraper API datasets, SERP parsing |
| Firecrawl | SDKs and REST | API key | Automatic | Managed | Markdown, JSON, schema extraction |
| Apify | Run an Actor with the client library | API token | Depends on the Actor | Depends on the Actor | Dataset items |
How the tools are compared
A scraper that works on a demo page tells you little about production. So the tools are judged on five criteria that matter once real traffic starts:
- Anti-bot and JavaScript handling: How well it gets through the blocks and renders the dynamic pages.
- Output quality: Whether you get structured, ready-to-use data or raw HTML you're going to have to parse it.
- Developer experience: Documentation, SDKs and how quickly you can get a first working request.
- Scale and cost predictability: How pricing, concurrency and credit multipliers behave as you scale.
- Flexibility and ecosystem: More products, more integrations and MCP support.
The better API is the one that keeps returning good data as the target tightens up and your volume increases.
The 6 tools at a glance
| Tool | Category | Best for | Coding knowledge needed |
|---|---|---|---|
| ScrapingBee | Request-in, HTML-out API | Simple, flexible single-call scraping | Beginner |
| Decodo | Request-in, HTML-out API | Template-based scraping on a proxy network | Beginner to intermediate |
| WebScrapingAPI | Request-in, HTML-out API | General-purpose scraping with residential proxies | Beginner |
| Bright Data | Scraping platform | Protected sites and large-scale data collection | Intermediate to advanced |
| Apify | Scraping platform | Ready-made scrapers and custom Actors | Beginner to advanced |
| Firecrawl | AI-native API | LLM-ready Markdown and JSON | Beginner to intermediate |
Scorecard against the five criteria
These ratings are purely the result of my editorial judgment based on what is documented as capability, not any measured test results.
| Tool | Anti-bot and JS | Output quality | Developer experience | Scale and cost predictability | Flexibility and ecosystem |
|---|---|---|---|---|---|
| ScrapingBee | Medium | High | High | High | Medium |
| Decodo | Medium | Medium | Medium | Medium | Medium |
| WebScrapingAPI | Medium | Medium | Medium | Medium | Low |
| Bright Data | High | High | Low | Medium | High |
| Apify | Medium | High | Medium | Low | High |
| Firecrawl | Medium | High | High | High | High |
The trade-off is clear. The request-in, HTML-out APIs are the simplest and most predictable, but you do more of the parsing. The platforms handle the hardest targets and deliver structured data, at the cost of complexity. Firecrawl sits in between, with clean output and a small API surface.
Who should pick what
| You are... | Start with | Why |
|---|---|---|
| A developer who wants one HTTP call that returns a page | ScrapingBee | Published credit costs, built-in rendering, optional extraction |
| A team scraping heavily protected sites at scale | Bright Data | Unblocking infrastructure is the core product |
| Building an LLM app, RAG pipeline, or agent | Firecrawl | Markdown and JSON output, plus crawl, search, and extract |
| Someone who needs a scraper for a specific site, fast | Apify | A store of ready-made Actors, plus a platform for your own |
| Already working with a proxy network and want templates | Decodo | Target templates and parsing on top of proxies |
| A developer who wants a general-purpose API with residential proxies | WebScrapingAPI | Rendering, proxy tiers, and extraction rules behind one endpoint |
My default picks: ScrapingBee as a general-purpose API, Bright Data when blocking is the main problem, Firecrawl for LLM applications, and Apify when a ready-made scraper already exists. Replace any of these if the specifics below fit your project better.
Category 1: Request-In, HTML-Out APIs
These are the simplest to adopt. You just call one endpoint with a URL and a couple of parameters,s and you get the page back. They're the best place to start if you already know how to parse HTML.
1. ScrapingBee

ScrapingBee is the API I'd hand to a dev who just wants a single HTTP call that returns a page. Its HTML API documentation lists a URL parameter, JavaScript rendering via a headless browser (on by default), a proxy ladder,r and the ability to extract data via CSS selectors or AI.
What makes it useful
- JavaScript rendering is on by default,t and you can turn it off for cheaper static fetches.
- A proxy ladder: begin with the default pool, add
premium_proxyif you get blocked, and then escalate tostealth_proxyfor the really nasty sites. -
extract_rulesreturns structured JSON from CSS selectors instead of HTML. -
ai_queryandai_extract_rulesextract data from a plain-language description or a JSON schema. -
js_scenariocan click, scroll, fill forms, and wait before the page is captured. - Screenshots, country targeting, and official SDKs.
Here is a basic request with CSS extraction. The docs recommend sending the key in an Authorization header, and parameters such as extract_rules must be stringified JSON.
import json
import os
import requests
API_KEY = os.environ["SCRAPINGBEE_API_KEY"]
response = requests.get(
"https://app.scrapingbee.com/api/v1/",
headers={"Authorization": f"Bearer {API_KEY}"},
params={
"url": "https://example.com/product/42",
"render_js": "true",
"extract_rules": json.dumps({"title": "h1", "price": ".price"}),
},
timeout=90,
)
response.raise_for_status()
print(response.json())
The important part is to climb the proxy ladder only when you need to, because each step costs more credits. This helper tries the cheapest option first:
TIERS = [{}, {"premium_proxy": "true"}, {"stealth_proxy": "true"}]
def fetch(url: str) -> requests.Response:
for extra in TIERS:
r = requests.get(
"https://app.scrapingbee.com/api/v1/",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"url": url, **extra},
timeout=90,
)
if r.status_code == 200:
return r
raise RuntimeError(f"Blocked at every tier: {url}")
For irregular pages, swap CSS rules for a natural-language query:
params = {
"url": "https://example.com/product/42",
"ai_query": "the product name, current price, and whether it is in stock",
}
Where it falls short: Credit multipliers add up. As at the time of writing the documentation lists a default Javascript rendered request at 5 credits, premium proxies with rendering at 25, stealth proxies at 75 and AI extraction as an additional 5. So check the current table before you go and work out your costs. If you don't have any extraction rules or AI you just get the HTML and you have to parse it yourself.
My verdict: ScrapingBee is my go to for developers looking for one flexible endpoint, with clear credit costs and an obvious escape hatch for the really hard sites.
2. Decodo

Decodo, formerly Smartproxy, offers a web scraping API on top of its proxy network. Its parameter reference describes a POST endpoint that takes a JSON body, with options for rendering, geography, device type, proxy tier, and parsing for supported targets.
What makes it useful
- JavaScript rendering through the
headlessparameter, with HTML for rendered pages and PNG for screenshots. - A
proxy_poolchoice betweenstandardfor simple pages andpremiumfor complex anti-bot measures. - Target templates, such as search engine results, with a
parseoption that returns structured data. - Geolocation, device type, custom headers and cookies, and session reuse of the same IP for up to ten minutes.
- Markdown output is available through some of its integrations, which suits LLM workflows.
A minimal Python call looks like this. The authorization value is the token from your Decodo dashboard.
import os
import requests
TOKEN = os.environ["DECODO_AUTH_TOKEN"]
response = requests.post(
"https://scraper-api.decodo.com/v2/scrape",
headers={
"Accept": "application/json",
"Content-Type": "application/json",
"Authorization": f"Basic {TOKEN}",
},
json={
"url": "https://example.com/products",
"headless": "html",
"geo": "United States",
},
timeout=90,
)
result = response.json()["results"][0]
print(result["status_code"])
print(result["content"][:300])
The important part is to turn on JavaScript rendering only when you need it. The docs note that rendering is slower and more resource-intensive, and that static sites don't need it.
Where it falls short: The documentation notes that the older Core and Advanced plans are no longer sold and points customers to an all-in-one Web Scraper API, so supported parameters can differ by plan and target template. Check the response shape and parameter list for the plan you actually have.
My verdict: Decodo is a good fit if you're after proxy-backed scraping with ready-made target templates and you're happy to read its parameter tables.
3. WebScrapingAPI

WebScrapingAPI is a general-purpose scraping API with rotating proxies and a headless browser all under one endpoint. The documentation covers rendering, JS, and extraction, and they have SDKs for Python, Node.js, and Scrapy.
What makes it useful
- A GET endpoint that takes
api_key,url, and optional parameters. -
render_jsto run a headless browser, plus wait conditions such aswait_untilfor dynamic pages. -
proxy_typeto choose between datacenter and residential IPs, andcountryfor geotargeting. -
js_instructionsto scroll or click before the HTML is captured, andextract_rulesfor structured output. - Official SDKs, including one for Scrapy, so it fits into existing crawlers.
import os
import requests
response = requests.get(
"https://api.webscrapingapi.com/v2",
params={
"api_key": os.environ["WEBSCRAPINGAPI_KEY"],
"url": "https://example.com/products",
"render_js": 1,
"proxy_type": "residential",
"country": "us",
"wait_until": "domcontentloaded",
},
timeout=90,
)
print(response.status_code)
html = response.text
Or with the official Python SDK, installed with pip install webscrapingapi:
from webscrapingapi import WebScrapingApiClient
client = WebScrapingApiClient(api_key="YOUR_API_KEY")
response = client.get("https://example.com/products", params={"render_js": 1})
Where it falls short: Rendering and residential proxies consume more credits per request than a basic fetch, so check the credit table for your plan. Its documentation has existed across multiple API versions, and defaults such as render_js can differ between them, so set the parameters you rely on explicitly.
My verdict: WebScrapingAPI is a solid general-purpose option for developers who want rendering and residential proxies behind a simple GET call.
Category 2: Scraping Platforms
These go beyond a single endpoint. They offer several products, from unblocking proxies to ready-made scrapers, and they suit teams with harder targets or bigger workloads.
4. Bright Data

Bright Data is less a single API than a web data platform. Its Web Unlocker documentation describes a REST endpoint that takes a zone name, a target URL, and a response format. Around it sit a SERP API, a Scraper API with prebuilt scrapers for specific sites, a Browser API for full browser sessions, proxy networks, datasets, and an MCP server for AI agents.
What makes it useful
- Web Unlocker handles blocks, CAPTCHAs, and retries behind one request.
- Prebuilt scrapers return structured data from popular sites, so you skip the parsing.
- A Browser API for pages that need a real browser session, compatible with Puppeteer and Playwright.
- A SERP API for structured search engine results.
- An MCP server that exposes these tools to Claude, Cursor, and other agents.
A Web Unlocker request is a single authenticated POST. The zone is the product configuration you create in the control panel.
import os
import requests
response = requests.post(
"https://api.brightdata.com/request",
headers={
"Authorization": f"Bearer {os.environ['BRIGHTDATA_API_KEY']}",
"Content-Type": "application/json",
},
json={
"zone": os.environ["BRIGHTDATA_UNLOCKER_ZONE"],
"url": "https://example.com/product/42",
"format": "raw",
},
timeout=120,
)
response.raise_for_status()
html = response.text
For sites covered by a prebuilt scraper, call the datasets endpoint instead and get structured JSON back:
response = requests.post(
"https://api.brightdata.com/datasets/v3/scrape",
params={"dataset_id": "YOUR_DATASET_ID", "format": "json"},
headers={"Authorization": f"Bearer {os.environ['BRIGHTDATA_API_KEY']}"},
json={"input": [{"url": "https://www.example.com/product/42"}]},
timeout=120,
)
print(response.json())
The important part is to match the product to the job. Bright Data's own guidance says not to drive Web Unlocker with Puppeteer, Playwright, or Selenium, and to use the Browser API when you need a browser. Other request options, such as country targeting and Markdown output, are documented in its API reference, so check the current options there.
Where it falls short: The breadth is also the complexity. You have to know your zones, your products, and your credentials before the first request, and pricing is product-specific. It's usage-based, which is very efficient for hard-to-reach sites and very expensive for easy ones. You're also still ultimately responsible for data collection in a legal and compliant manner with respect to each site's terms.
My verdict: Bright Data is the strongest option when getting blocked, not parsing, is your main problem, and you have the engineering time to learn a larger platform.
5. Apify

Apify is a platform for running Actors, which are serverless programs that scrape websites, process data, and automate workflows. You can run one of the many ready-made Actors in the Apify Store, or build and host your own. The Python client quick start shows how to call them from code.
What makes it useful
- A store of ready-made Actors for specific sites and tasks, so you often don't need to write a scraper.
- A place to deploy your own scrapers, with scheduling, storage, and logs.
- Official clients for Python and JavaScript, a REST API, and a CLI.
- Datasets that hold structured results, readable page by page.
- Crawlee, Apify's open-source crawling library, plus an MCP server for AI agents.
Running an existing Actor is three steps: authenticate, start the run, and read the dataset. Each Actor defines its own input schema, so check its page for the fields it expects.
# pip install apify-client
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {"startUrls": [{"url": "https://example.com"}]} # varies per Actor
run = client.actor("username/actor-name").call(run_input=run_input)
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
The same Actor is available from the command line, which suits scripts and cron jobs:
echo '{"startUrls": [{"url": "https://example.com"}]}' | apify call username/actor-name --silent --output-dataset
The key thing is to look at the Actor not the platform. Store Actors are created by all sorts of authors so you need to check their ratings, their success rate and their maintenance history before you go and use one.
Where it falls short: Pricing is not uniform. Actor pages describe different models, such as paying for platform usage or paying per event or result, so your cost depends on which Actor you pick. The client library's return types have also changed across versions, so check the current quick start if the example above doesn't match your install.
My verdict: Apify is the best choice when a ready-made scraper already exists for your target, or when you want one place to host your own scrapers with scheduling and storage.
Category 3: AI-Native API
This group returns data in a format that's language-model-shaped rather than raw HTML.
6. Firecrawl

Firecrawl is built for developers who want web data in a form that LLMs can consume. It's a single API that does scraping, crawling, mapping, search, and structured extraction and returns clean Markdown or JSON. See the documentation for the current endpoints, including interacting with pages and monitoring them for changes.
What makes it useful
- Scrape one page, crawl a whole site, map its URLs, or search and scrape results.
- LLM-ready Markdown, or structured JSON from a schema or a prompt.
- SDKs for Python, Node.js, Go, Rust, and Java, plus a CLI.
- A hosted MCP server, so Claude, Cursor, and other MCP clients can call it directly.
- An open-source version you can self-host.
Response field names have varied between SDK versions, so check the current SDK reference before copying this into production.
# pip install firecrawl-py
import os
from firecrawl import Firecrawl
app = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
# Scrape one page to LLM-ready Markdown
doc = app.scrape("https://example.com/pricing", formats=["markdown"])
print(doc.markdown[:300])
# Crawl a documentation site
crawl = app.crawl(
"https://docs.example.com",
limit=50,
scrape_options={"formats": ["markdown"]},
)
for page in crawl.data:
print(page.markdown[:100])
For structured output, pass a schema so the response has a predictable shape:
from pydantic import BaseModel
class Product(BaseModel):
name: str
price: float
in_stock: bool
result = app.extract(
urls=["https://example.com/product/42"],
schema=Product,
system_prompt="Extract the product name, price, and availability",
)
print(result.data)
Where it falls short: it'scredit-based,d with one credit covering a basic page scrape/crawl/map; crawling large sites will quickly use up your credits. More intensive features like extraction will use more credits, and if you're planning on self-hosting, you'll need to check what the open-source version includes versus the hosted version.
My verdict: Firecrawl is my go-to for developers building LLM apps, RAG pipelines,s and agents that need clean web content with minimal setup.
Pricing, scale, and lock-in
Prices change too often to list, but the models differ in ways that matter:
- Credit-based APIs (ScrapingBee, WebScrapingAPI, Firecrawl, etc.) all charge you different rates depending on the options you choose. Rendering, premium/residential proxies, and AI extraction are all options that will increase the cost of a request.
- Usage-based platforms such as Bright Data and Apify price by product, volume, or Actor, so estimate cost per successful page, not per request.
- Plan-based services such as Decodo have changed their plan structures over time, so confirm which parameters and limits apply to the plan you buy.
Lock-in is low for simple APIs. Url+params services are all pretty much the same so you can just have a thin adapter to switch providers. It's higher with Apify Actors and prebuilt datasets where your code is tied to a specific scraper and its output format.
In general, the more the service does for you, the more you should check about export options and exit costs before you commit.
Production patterns that apply to any scraping API
These patterns wrap whichever service you choose, and they catch most real-world failures.
Detect block pages
A 200 response can still be a CAPTCHA or an access denied page. Always look for the expected content before you go around treating a response as a success.
MARKERS = ("captcha", "access denied", "verify you are human")
def looks_blocked(html: str) -> bool:
text = html.lower()
return len(html) < 500 or any(marker in text for marker in MARKERS)
Retry with exponential backoff.
Retry transient errors and rate limits with growing delays, and stop after a few attempts. Check whether your provider bills failed requests, because retries can cost credits.
import random
import time
import requests
RETRYABLE = {429, 500, 502, 503, 504}
def get_with_retries(url, attempts=4, base=1.0, **kwargs):
for attempt in range(attempts):
response = requests.get(url, timeout=90, **kwargs)
if response.status_code not in RETRYABLE:
return response
time.sleep(base * 2 ** attempt + random.random())
response.raise_for_status()
return response
Cap concurrency at your plan's limit
The majority of services will limit you by plan in terms of the number of concurrent requests you can make. A semaphore will keep you under that limit so that you won't get errors from a burst.
import asyncio
import httpx
LIMIT = 10 # stay at or below your plan's concurrency
async def fetch_all(urls, make_request):
semaphore = asyncio.Semaphore(LIMIT)
async with httpx.AsyncClient(timeout=90) as client:
async def one(url):
async with semaphore:
return await make_request(client, url)
return await asyncio.gather(*(one(u) for u in urls), return_exceptions=True)
Fallback across providers.
Because the simple APIs share a shape, a small adapter lets you try a cheap provider first and escalate to a stronger one only when it fails.
from typing import Callable, Optional
Fetcher = Callable[[str], Optional[str]] # returns HTML, or None on failure
def fetch_with_fallback(url: str, fetchers: list[Fetcher]) -> Optional[str]:
for fetch in fetchers:
try:
html = fetch(url)
except Exception:
html = None
if html and not looks_blocked(html):
return html
return None
Run your own comparison.
The scores in this guide are derived from the documentation, not from your actual targets. You'll need to test it yourself. Write one small function per service that takes a URL as input and returns the HTML. Run those through the same harness, and it will report the success rate, the median and p95 latency, and the effective cost per 1,000 good pages (accounting for failures).
import time
from statistics import median
def benchmark(name, fetch, urls, expected_marker, cost_per_request):
latencies, successes = [], 0
for url in urls:
start = time.perf_counter()
try:
html = fetch(url)
except Exception:
html = ""
latencies.append(time.perf_counter() - start)
if html and expected_marker in html and not looks_blocked(html):
successes += 1
rate = successes / len(urls)
cost_per_1000 = cost_per_request * 1000 / rate if rate else float("inf")
p95 = sorted(latencies)[int(0.95 * (len(latencies) - 1))]
print(
f"{name}: success {rate:.0%}, median {median(latencies):.1f}s, "
f"p95 {p95:.1f}s, about ${cost_per_1000:.2f} per 1,000 good pages"
)
# benchmark("scrapingbee", scrapingbee_fetch, URLS, "Add to cart", 0.0005)
Use at least 50 URLs per target site (including your toughest pages) and run them at various times of day. Block rates are time dependent, target dependent and IP reputation dependent.
What scraping APIs can't replace
A scraping API removes the infrastructure work, but it doesn't remove the responsibility.
-
Rules and permissions still matter: Check the site's terms and
robots.txt, don't collect any personal data that you don't have a right to, and get legal advice if in any doubt. Just because a service can fetch a page doesn't mean you can use the data. - Success rates depend on the target: A service that works a treat on one site can be a total failure on another. You have to test it on the actual pages you're going to be using, not on a demo URL.
- Extraction needs verification: CSS rules will break if the markup changes and AI extraction can misread a field. Outputs must be validated against a schema.
- Cost has to be modeled: Rendering, premium proxies, retries, and AI all eat into your credits. You need to work out your cost per successful page before you start to scale.
Scraping APIs work best when they're doing the fetching, and you're then responsible for what you actually collect, how you use it, and whether it's correct.
Final verdict
No one API works for every project.
- If you're after one flexible endpoint then ScrapingBee is the one to go for.
- If you're up against heavy anti-bot protection at scale then Bright Data is the API for you.
- If you're building with LLMs then Firecrawl is the way to go.
- If there's a ready made scraper available for your target then use Apify.
- If you're after templates on a proxy network then go for Decodo.
- For general purpose rendering with residential proxies, go with WebScrapingAPI.
A scraping API obviously cuts the time between wanting the data and getting it, but the first successful request is only the start. You've got to pick the service whose output, pricing, and failure characteristics still make sense as you scale up and as your targets get more sophisticated.
Frequently asked questions
What is the best web scraping API?
It depends on the job. ScrapingBee is good for simple, flexible single-call scraping. Bright Data is the one for protected sites at scale. Firecrawl is the one for LLM and RAG use cases. Apify is for ready-made scrapers. Decodo and WebScrapingAPI are the ones for general-purpose scraping on proxy networks.
Do I need my own proxies if I use a scraping API?
No, they route the requests through their own proxy pools. You pick a tier (standard, premium, residential, stealth, etc.) rather than having to manage your own IPs.
When should I use a scraping API instead of Playwright or Puppeteer?
Use an API when you're dealing with blocking, proxy management, or scaling issues. Use your own browser automation if you need full control over the interactions and can manage the infrastructure yourself.
Is web scraping legal?
It all depends on the site, the data, and where you are. Check the site's terms and robots.txt. Don't go collecting personal data unless you have a proper legal basis for doing so. If you're doing it for a commercial project or one that involves sensitive data, then you absolutely need to get legal advice.
How do credits work?
The majority of services will charge you a base rate per request and then scale up for things like JS rendering, premium/residential proxies and AI extraction. Always check the current credit table and work out your cost per successful page rather than per request.
Top comments (22)
The line that stuck with me is that the best API is the one whose pricing and failure mode still make sense at 10x your starting volume. Most comparisons stop at “which one worked on my demo URL.”
I also appreciated that you said upfront this is an editorial comparison based on docs, not a controlled benchmark. That’s rare in this category, and the “run your own comparison” harness at the end makes up for it. Measuring cost per 1,000 good pages instead of per request is the right way to compare, since retries and block pages quietly eat credits.
The
looks_blocked()check is a small thing that catches a lot. A 200 response that’s really a CAPTCHA page has burned me before.One thing I’d love to see in a follow-up is how these behave on heavily JS-driven pages with infinite scroll or login walls. ScrapingBee’s
js_scenarioand Bright Data’s Browser API seem like the interesting ones there, and I’m curious whether Firecrawl’s page interaction holds up in practice.Bookmarked the fallback adapter pattern too. Starting cheap and escalating only on failure is a nice way to keep costs predictable.
I’m glad the cost-per-good-page framing was useful. That’s the number that matters in practice, and it’s the one most comparisons skip.
You’re right about the block-page check. A 200 that’s really a CAPTCHA wall is the sneakiest failure because nothing errors out, the bad data just flows downstream.
Your follow-up question is a good one, and I should be upfront that this guide is based on documented capabilities, not hands-on tests of infinite scroll or login flows, so I can’t tell you how Firecrawl’s page interaction holds up there. What the docs suggest:
js_scenariocan scroll, click, fill forms, and wait before capture, so it’s the most direct fit for infinite scroll through a single call.A follow-up focused on JS-heavy and authenticated pages, with real tests on the same URLs across providers, is a great idea. Thanks for the suggestion. Also worth remembering that scraping behind a login often raises extra terms-of-service questions, so check those first.
Your 10x volume point is the one I’d stress too. A tool that’s cheap at 5,000 pages can behave very differently at 500,000, when retries, concurrency limits, and block rates start to dominate the bill.
On JS-heavy and authenticated pages, I agree with the split you described. ScrapingBee’s
js_scenariolooks like the simplest route for infinite scroll, and Bright Data’s Browser API looks stronger for login flows and multi-step sessions. I haven’t tested either on those cases, so I’d rather not claim more than the docs say.A follow-up with real tests on the same JS-heavy URLs across providers is on my list, with the harness extended to cover scroll depth and session handling. I’ll tag you when it’s up.
Thanks, and I'm glad the 10x point resonates. At 500,000 pages, retries, concurrency caps, and block rates are what you end up paying for, so the headline per-request price tells you little.
I'm also glad you're not claiming more than the docs support. The js_scenario versus Browser API split in my post comes from documented capabilities, not from tests, so I'd hold your impressions to the same standard until you've run them.
A few things that might be worth building into the follow-up harness:
Please do tag me when it's up. Real numbers on the same URLs across providers would be more useful than anything in my post.
That’s a useful way to frame it, and it matches how I’d approach the follow-up too.
The distinction between “can do it” and “does it reliably” matters most on infinite scroll and login walls. A single js_scenario call that scrolls is easy to describe in docs, but the real test is how consistently it triggers lazy loading, how it handles rate limits mid-session, and what a partial load looks like when nothing errors out. That’s the same silent-failure problem as the 200 CAPTCHA page, just with half-loaded content.
For the follow-up, I’d measure the same cost per 1,000 good pages metric, but define “good” more strictly for JS-heavy targets, for example by checking that the item count after scrolling meets a minimum threshold. Running identical URLs across providers would make the comparison fair.
Appreciate the thoughtful suggestion. If you have a specific infinite-scroll or authenticated target in mind, I’d be glad to include a similar one in the test set.
Thanks, this is exactly the gap I’d like the follow-up to cover. The post is an editorial comparison built from documented capabilities, not a controlled benchmark, so “can do it” versus “does it reliably” is where a real test would add the most.
I agree on infinite scroll and login walls. A scroll instruction like
js_scenarioorjs_instructionsis easy to describe, but the failure that matters is a partial load that returns a 200 with half the items. It’s the same silent-failure problem as the CAPTCHA page that passes a status check.For the follow-up I’d keep cost per 1,000 good pages as the headline metric, but tighten “good” for JS-heavy targets. Each page would need a minimum item count after scrolling, plus the expected marker and the block-page check from the harness in the post. Every provider would get the same URLs, run at different times of day.
If you have an infinite-scroll or authenticated target in mind, send it over and I’ll consider adding it to the test set.
Rolling your own with BeautifulSoup is an option as well
Fair point, and for the right targets it's the cheapest option. For static pages with no aggressive bot protection,
requestsplus BeautifulSoup costs next to nothing and gives you full control.One distinction worth making is that BeautifulSoup only parses HTML. It doesn't fetch pages, render JavaScript, rotate proxies, or deal with CAPTCHAs, so the hard part is everything around it. That cost grows with volume: at 100 pages you barely notice it, but at 100,000 you're maintaining proxy pools, retry logic, and headless browsers yourself.
It also isn't either/or. The HTML-out APIs in the post (ScrapingBee, Decodo, WebScrapingAPI) return raw HTML, so you can still parse it with BeautifulSoup and let the API handle fetching, rendering, and blocks. A common middle path is plain
requestsfor easy domains and an API only for the ones that block you, which fits the tier-routing idea from earlier in this thread.I touch on this in the FAQ on APIs versus Playwright or Puppeteer, but I should have said more about the DIY route. Thanks for raising it.
The five-criteria scorecard is useful, but the part I’d screenshot is the “what scraping APIs can’t replace” section. People tend to forget that a service can fetch a page doesn’t mean you’re allowed to use the data. Checking
robots.txtand the site’s terms before you scale is cheap insurance.I also like that the table comparing request styles shows how similar the simple APIs are. That’s what makes the thin-adapter and provider-fallback idea so practical. You can start on the cheapest option and only escalate when a URL actually fails, instead of paying stealth-tier prices for every request.
Two things I’d love to see in a follow-up: a real side-by-side run on the same 50 URLs, using your own benchmark function, with success rate and cost per 1,000 good pages for each tool, and a note on how each one handles pages that need login or heavy scrolling. That’s where I’ve seen most of these services behave very differently.
Glad the “what scraping APIs can’t replace” section landed. You’re right that fetching a page and being allowed to use its data are separate questions, and I’d rather people hit that early than after they’ve built a pipeline on top of it.
You also read the fallback idea the way I meant it. The simple APIs take a URL and a few params and return HTML, so a thin adapter is cheap to write. Start on the cheap tier and escalate only when
looks_blockedfires. For ScrapingBee that’s the difference between paying 5 credits and paying 75 on every request. That’s also why I’d model cost per successful page rather than per request.I should be upfront about the limits of the piece. The scorecard is my editorial read of the documentation, not measured results, and I said so in the article. A real side-by-side on the same 50 URLs is the follow-up I’d most like to write, and the
benchmarkfunction is already there for it. I’d report success rate, median and p95 latency, and cost per 1,000 good pages, since failed requests and retries can eat credits. I’d run it at different times of day, because block rates drift with IP reputation. I’d also include at least one site that’s hostile to scrapers, so the numbers don’t just reflect easy targets.On login-gated pages and heavy scrolling, I only touched on this by listing the relevant parameters
(js_scenarioin ScrapingBee,js_instructionsin WebScrapingAPI, the Browser API in Bright Data) without testing them. That’s a real gap, and I agree it’s where these services diverge most. I’d test three cases:I’d also note which of these are even permitted under a site’s terms, which ties back to your first point.
If you’ve seen specific services fail on login or scroll, I’d like to hear which ones and what the failure looked like. I’ll fold real cases into the follow-up, with credit. Thanks for the thoughtful read.
The real test of the APIS is which one's pricing and failure mode still make sense at 10x volume. That's the line that really cuts it. Most of the other comparisons just stop at' which one works on the demo page”.
I also liked that you made it very clear at the outset that this is an editorial comparison rather than a controlled benchmark. That makes the scorecard a lot more credible. The block/page detection bit is a good one as well in that a 200 response which is in fact a CAPTCHA page is the sort of thing that silently kills a scraped dataset.
The proxy ladder helper (default, then premium, then stealth) is a really good way to keep your credit costs down. One thing I’d add is that when you're moving up the tiers you should log which URLs required which tier. After a week you can then take the known hard domains and just route them straight to the more expensive proxy, thereby eliminating the wasted attempts on the cheaper proxies.
One thing I'm curious about regarding the Firecrawl vs ScrapingBee
ai_querystyle extraction is whether there's much drift on the irregular pages (e.g. a price field occasionally picking up the crossed out old price). I’d like to know if schema validation on its own is sufficient to catch that or whether you actually needed to do some spot checks.Bookmarking this for the “cost per 1,000 good pages” figure on the harness.
Thanks, you picked out the parts I care most about. Most comparisons stop at “does it work on the demo page,” and what matters in practice is whether the pricing and failure behavior still hold up at 10x volume. That’s why I framed the scorecard as an editorial comparison and not a benchmark. I’d rather be upfront about what it can and can’t claim.
Your tier-logging idea is a good addition. Record which URLs needed which tier, then after a week pin the known hard domains straight to the expensive tier. That removes the failed cheap attempts, and those failed attempts are where the credits quietly drain. A per-domain “minimum working tier” map with a periodic re-test, so you can step back down if a site loosens up, would make it even better.
On the
ai_query-style extraction drift: I wouldn’t say schema validation alone is enough. It catches type and shape problems, like a missing field or a string where a number should be. It won’t catch a price field that is valid but wrong, such as the crossed-out old price. For irregular pages I’d add cheap sanity checks on top of the schema: compare the extracted price against a plausible range for that category, flag cases where the “current” price is higher than the “original” one, and spot-check a small random sample by hand each run. The sample is what tells you if drift is creeping in over time.Glad the “cost per 1,000 good pages” figure is useful to you. Counting only pages that pass block detection and validation, rather than all requests sent, is what makes the cost comparison honest.
Useful comparison, thanks for testing them side by side. One thing readers often overlook: the cheapest option per request isn't always the cheapest per usable result, so it's worth testing on your own target site before committing.
Thanks for reading! One clarification: this is an editorial comparison based on each provider's documented capabilities, not a side-by-side benchmark on the same pages. That's exactly why I agree with your point. Cost per request and cost per usable result can differ a lot, especially once rendering, premium proxies, and retries kick in. I included a small benchmark harness near the end that reports success rate, p95 latency, and cost per 1,000 good pages, so you can run it against your own target sites before committing.
The “Who should pick what” table is the most useful part for me. I’d have saved a lot of time if I’d had it when I started a scraping project last year.
I went the Playwright-plus-my-own-proxies route first and spent more time fighting retries and IP bans than actually working with the data. Moving to an API cut that maintenance a lot, though I did have to get used to paying per request.
Your FAQ answer on APIs vs. Playwright is right, but I’d add one thing. For small, low-volume jobs on friendly sites, plain Playwright or even
requests+ BeautifulSoup is often still the cheapest option. The APIs start to pay off once blocking or scale becomes the real problem.The Firecrawl section also made me curious about the self-hosted version. Has anyone here run it in production? I’d like to know how it compares to the hosted one in output quality and how much effort it takes to maintain.
Bookmarked for the benchmark script alone. Running 50+ URLs per target at different times of day is advice more people should follow.
The three-category split (HTML-out APIs, platforms, AI-native) made the landscape easier to follow than most “top 10 scrapers” lists, which just dump everything into one pile.
The comparison table of request styles, auth, and proxy tiers was my favorite part. It shows how similar the simple APIs really are, so switching providers is less scary than it looks. The thin-adapter-plus-fallback pattern follows from that, and I’m going to try it on a side project.
The ScrapingBee credit multipliers (5 for rendering, 25 for premium proxies, 75 for stealth) are a good reminder to model cost per successful page. A “cheap” plan can get expensive fast on JS-heavy sites.
Two questions:
Thanks for the honest “check current docs” disclaimer too. Pricing in this space changes constantly.
The request-style table was the most useful part for me, because seeing
render_js,proxy_type, andextract_rulesside by side shows how little code it takes to switch providers.That also supports your thin adapter idea. I’d go one step further and keep a per-provider config map (parameter names, auth style, credit cost per option) so the fallback function can swap vendors without touching the calling code. Lock-in really only starts when you adopt something like Apify Actors or a vendor’s dataset format.
The legal section deserves more attention than these posts usually give it. “The service can fetch it” and “I’m allowed to use it” are very different questions, and a lot of people only find that out after the data is already in production.
One follow-up request: a real run of your benchmark harness against the same 50 or so URLs, including a couple of tough e-commerce pages. Even a small table of success rate and cost per 1,000 good pages would turn this from a good guide into the reference people link to.
Agreed on the config map. I’d add response normalization to it, since the parameter names are the easy part. Providers differ more in what “success” means: some return a 200 with a captcha page, some bill you for failed JS renders, and the error shapes are all different. If the adapter also maps each vendor’s output into one schema and runs a “did we get the real page” check, switching vendors really stays a config change.
That check is also what makes the benchmark request workable. “Cost per 1,000 good pages” only means something if “good” is defined by a validator, like a selector or price field that must be present, and not just an HTTP 200. Otherwise the success rates on tough e-commerce pages look better than they are.
Good call on the legal section too. Scraping public pages, storing personal data and reusing content commercially each carry different risks, and they’re easy to blur together.
A few technical notes from running scrapers in production that pair well with your patterns:
looks_blockedcheck is a good start, but I’d go one step further and assert on a structural marker (a selector or JSON-LD block that must exist) before accepting a response. Soft blocks often return 200 with a full-length HTML page that’s just a login wall or a “was this you?” interstitial, so length checks alone miss them.Great point about the thin adapter for provider fallback. Lock-in really is low for URL+params APIs, and that’s the biggest advantage over Actor-based setups.
The part that stuck with me is the line about choosing the API whose pricing and failure mode still make sense at ten times your volume. Most roundups only compare features, but cost per successful page is what actually bites once you scale.
The block-page check was useful too. A 200 response that turns out to be a CAPTCHA page is one of those things you only learn after your dataset is full of junk. Pairing it with the ScrapingBee proxy ladder helper makes sense, since escalating tiers only when needed keeps the credit multipliers from eating the budget.
I also appreciated that you said upfront this is an editorial comparison and not a controlled benchmark. It makes the scorecard easier to trust, and the 50 URLs per target benchmark harness is a good way to test it on real pages.
Have you run into a site where the cheap tier looked fine in testing but fell apart at volume? I’m curious whether Firecrawl’s credit usage stays manageable on large crawls.