Why your scraper gets blocked
Anti-bot systems score a request on several signals at once, and the network is the cheapest one for them to check. A datacenter IP announces itself in the ASN lookup before your headers are even parsed — Cloudflare, DataDome and PerimeterX all keep lists of hosting ranges, and a request from one starts with a penalty no amount of header tuning removes.
Residential addresses come from real home connections, so the same lookup returns Comcast or Deutsche Telekom rather than AWS. That does not make you invisible — request rate, TLS fingerprint and behaviour still count — but it removes the signal that gets you blocked before anything else is considered.
How many proxies you actually need
This is the question people get most wrong, usually because they are thinking in the per-IP pricing of a different product. With rotating residential you do not buy IPs at all; you buy traffic, and every request can leave from a different address at no extra cost.
What you size instead is concurrency and rate. A polite scraper at two requests per second against one site needs no pool planning whatsoever. A hundred concurrent workers across many targets is still one credential — there is no per-session charge and no concurrency limit on our plans.
What a scrape costs, in gigabytes
Price per GB is meaningless until you know how many gigabytes your job is. Measure it rather than guess: fetch a hundred pages, divide the bytes by a hundred, multiply by your target.
| What you are fetching | Typical page | 1M pages | At $0.70/GB |
|---|---|---|---|
| Plain HTML, no assets | 30 KB | ~30 GB | ~$21 |
| Product pages, typical shop | 80 KB | ~80 GB | ~$56 |
| Search results pages | 150 KB | ~150 GB | ~$105 |
| Headless browser, assets loaded | 1.5–3 MB | 1.5–3 TB | $1,050+ |
The last row is the one that surprises people. Running a full browser and letting it load images, fonts and analytics multiplies your bill by twenty or more. Blocking non-essential resources in Playwright or Puppeteer is usually the single largest saving available to a scraping budget.
Rotating or sticky sessions
Rotating gives you a new address per request, which is what you want for independent page fetches. Sticky holds one address for up to thirty minutes, which is what you need whenever state has to survive between requests — a login, a cart, a paginated result set tied to a session cookie.
Both come from the same plan and the same credential; the difference is one field when you generate the line, so you can use each where it fits rather than choosing once.
Five things that cut blocks more than anything else
- Slow down before you scale up. Most blocks are rate, not detection. A scraper at one request per second per target runs for months; the same job at fifty runs for an hour.
- Match the geography to the content. Fetching a German shop through a Brazilian address gets you a different page and sometimes a challenge. Pick the country you are pretending to be from.
- Block assets in headless browsers. Images and fonts are most of your traffic and none of your data.
- Keep sessions sticky where state exists. An IP change mid-login looks exactly like a stolen cookie, because that is what it usually is.
- Retry on a new address, not immediately. A block answered by an instant retry from the same place confirms you are automated.
Common questions
How many proxies do I need for web scraping?
With rotating residential, none in the counted sense — you buy traffic, not addresses, and each request can leave from a different IP at no extra cost. Size the job by gigabytes and by how fast you intend to go, not by pool size.
Are datacenter proxies good enough for scraping?
For sites with no anti-bot, yes, and they are cheaper. Anything behind Cloudflare, DataDome or PerimeterX identifies hosting ranges immediately, and no header tuning gets around it.
How much traffic does scraping a million pages use?
Roughly 30–80 GB for plain HTML at 30–80 KB per page. A headless browser loading images and fonts uses 1.5–3 MB per page instead, which is 1.5–3 TB for the same million — blocking assets is the biggest saving available.
Will proxies alone stop me being blocked?
No. They remove the network signal, which is the one checked first and hardest to fix any other way. Request rate, TLS fingerprint and behaviour still matter, and a fast enough scraper gets blocked from any address.
Try it before you pay for it
Connect Telegram and we credit 1 GB to your account — enough to measure your own traffic use on your own targets rather than trusting the numbers above. After that it is $0.70–$2.00 per gigabyte depending on volume, traffic never expires, and there is no monthly fee.