Web scraping for beginners: fewer blocks by being a polite visitor

In plain English: web scraping means using a program to read web pages and pick out the information you want, such as prices or headlines, instead of copying it by hand. When a beginner's scraper gets blocked, the proxy is rarely the main reason. Websites block traffic that behaves like a clumsy robot: too fast, too bare, too repetitive. Change the behaviour and most of the blocks go away. Everything here assumes you're collecting public information and following each site's own rules.

Before you start: the ground rules

• Check whether the site offers an official API (an approved way for programs to get its data). If it does, use that instead of scraping.
• Read the site's terms of use and its robots.txt file, which lists the parts of the site its owner would like automated visitors to leave alone.
• Collect public information only, and leave personal data alone.

Step 1: match the proxy to the site

A simple site with little protection is fine on datacenter. Sites that sit behind bot-checking services such as Cloudflare or Akamai usually need residential. Apps built for phones may need mobile 4G/5G. Our guide to the three types explains the differences.

Step 2: send the headers a browser would send

Every request carries headers: short lines of information about who's asking and what they can accept. A real browser sends ten or more. A first script often sends almost nothing and introduces itself as User-Agent: python-requests/2.31.0, and plenty of sites answer that with an error or a stripped-down page. Sending the normal set, copied from your own browser, gets you the same page a visitor would see:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Upgrade-Insecure-Requests: 1

For a small personal project, it's also good manners to name your scraper honestly in the User-Agent and add a contact address, so a site owner can get in touch rather than simply blocking you.

Step 3: keep one identity per session

The User-Agent is the header that names your browser. Don't change it on every request from the same address; a visitor whose browser changes every few seconds makes no sense to a website. Pick one and keep it for as long as you keep that address.

Step 4: slow right down

Nobody reads 100 pages a second. Add a random pause between requests (0.5-2 seconds is a sensible start). If the site replies with a 429 ("too many requests") or a 503 ("service unavailable"), stop and wait. When the reply includes a Retry-After header, it tells you how long. Each time you're asked to slow down, wait longer than the time before; this habit is called exponential backoff.

Step 5: use a real browser only when you have to

Some sites build their pages with JavaScript (code that runs inside the browser), so a simple script receives an almost empty page. For those, you need a tool that drives a real browser:

• Playwright (Python or Node.js): a good first choice today. Our Playwright proxy guide covers the setup.
• Puppeteer (Node.js): dependable, and built around Chrome.
• Selenium: the oldest of the three, and still widely used.

A real browser uses far more data than a simple request, because it loads images and scripts too. On a per-GB plan, switch off the images you don't need.

Step 6: treat a CAPTCHA as a stop sign

A CAPTCHA (the "prove you're human" puzzle) means the site has decided your traffic looks automated. Don't try to get round it. Slow down, look at what you're sending, and check whether an official API would do the job. If a site keeps challenging a slow, polite scraper, take that as its answer.

Step 7: keep a simple log

Write down every blocked request: which address, which page and which reply code. If blocks suddenly jump, something has changed. Perhaps the site was updated, your pace crept up, or an address went bad. A small log makes the cause easy to spot.

Is this allowed?

Collecting public information is common, but it isn't automatically allowed everywhere. A site's terms may forbid it, and data protection and copyright law still apply to what you collect and how you use it, especially anything about people. Go slowly, stick to public pages, follow the site's rules, and think twice before republishing what you gather. When in doubt, ask the site owner.