Proxy for Scrapy: a beginner's guide
Learning Scrapy for a small project? What a proxy adds to your first spider, what it costs, and how to keep a hobby crawl polite and light on data.
Start here
Scrapy for beginners: a first spider and where the proxy setting goes
In plain English, Scrapy is a free Python tool that visits web pages and pulls out the bits you ask for. It has no settings screen, so the proxy goes in your code or your terminal. Quick start: install it with pip, create a project with scrapy startproject, then practise on the toscrape practice sites, which exist for learning, before adding any proxy.
Two ways to add the proxy
The simplest is setting the http_proxy and https_proxy environment variables in your terminal before running scrapy crawl; Scrapy uses them for every request. Or give a request a proxy meta value such as http://user:pass@host:port inside your spider.
Choose HTTP, not SOCKS5
Scrapy's standard downloader works with HTTP and HTTPS proxies; SOCKS5 needs an optional download handler that its own docs call experimental. So when you copy your proxy details, use the HTTP version and leave SOCKS5 for other apps.
Passwords with odd characters
If your proxy password contains symbols such as @, : or #, run it through Python's urllib.parse.quote before building the proxy URL. Otherwise Scrapy misreads where the password ends and the host begins, and the errors look like a dead proxy.
Polite habits Scrapy helps with
New projects come with ROBOTSTXT_OBEY switched on, so Scrapy skips pages a site's robots.txt rules out; leave it on. Set DOWNLOAD_DELAY to a second or two in settings.py, and stop the crawl if 429 errors, meaning too many requests, start appearing.
Quick start in three steps
Nothing to install. This is the whole route, from paying to the first page you load through the proxy.
Buy a small Datacenter plan
On the order form, pick a country, a protocol and a login method, then pay. At $2.49 per GB (gigabyte of data) it costs less than our other types, so a first test is cheap.
Paste it into the tool
Most tools have a proxy field or a list to paste into, often one proxy per line. Add a single proxy first and run one small task.
Check that it worked
Paste the details into our proxy checker: it shows whether the proxy answers and which IP address sites will see. The IP location lookup then tells you which country that address is in, and if anything looks wrong, the quick fixes further down cover the usual slips. Address right, but one site still turns you away? The last of the fixes below covers that.
Our pick for Scrapy: Datacenter
Scrapy is a free Python framework for collecting information from websites, and a spider is the small program you write with it. For a first project on public pages, datacenter is the type to start with: it costs less per GB than our other types and answers quickly. Scrapy also fetches only the pages you ask for, not the images and scripts a browser would load, so a small crawl stays light on data.
Made for concurrency
Concurrency means Scrapy fetches several pages at once instead of one after another. That's what makes it quick, and also what lets a beginner overwhelm a small site by accident. Switch on AutoThrottle, Scrapy's built-in speed regulator, and it adjusts the pace to how quickly the site is responding.
Drop-in middleware
Middleware is a small add-on that adjusts every request on its way out. Scrapy's proxy middleware comes built in and already switched on, so there's no extra package to install or configure: give it a proxy address and it handles the connection for you.
Rotate to ease past rate limits
Rotating gives each request a different address; sticky keeps one for a while, which suits pages that remember you, such as a basket. Rotation is no way round a site's limits, though. If a site starts answering 'too many requests', the fix is to slow down, not to add addresses.
Crawls aimed at a region
Some sites change prices, language or stock depending on where the visitor is. Pick a proxy country in your dashboard, and a city where one's offered, and your spider sees the version a local visitor would. That's handy for a small comparison project, as long as you stick to public pages.
What a proxy won't do here
It won't make a tool acceptable to a site or game that bans automation, and it can't stop them acting on an account the tool runs.
What people use it for
- ▸A first spider for a hobby or study project
- ▸Price checks for a one-person online shop
- ▸Seeing another country's version of a site
- ▸Practising polite crawling on public pages
Which proxy type should I pick for Scrapy?
The usual rule is to start with the type that costs least and step up only when a site says no. Here, datacenter is enough to begin with.
| Type | Choose it when… | From, per GB |
|---|---|---|
| Datacenter Our pick for Scrapy | Cost matters most and the site is happy with addresses from server companies. Good for public pages and first experiments. | $2.49 |
| Residential | A site turns server addresses away, or you want to see it the way a local visitor does. The address belongs to a home broadband line. | $3.98 |
| Mobile 4G/5G | The service is built around phones, or you want to see it as someone on 4G or 5G would. The address comes from a phone network. | $4.98 |
| Residential Premium | Standard residential keeps being refused on one particular site. Hand-picked home addresses, at a higher price per GB. | $6.99 |
Choosing a country: Pick the country your task needs and keep each job in one country, so what the tool sees stays consistent from run to run. There are 55+ countries, and in major markets some types let you pick a city too. Browse locations
For the longer version, our proxy comparison walks through all four types.
What you need to add a proxy to your software
Wherever you set it up, you type in the same four details from your dashboard: the host (the proxy's address), the port (a number that tells the connection which door to use), a username and a password.
Inside an app, tool or script
Some apps and most automation tools have a proxy box of their own, which keeps the change to that one program. In code, the proxy is usually a single setting.
Two ways to log in to the proxy
- ▸Username and password. The
host:port:user:passdetails that browsers, apps and scripts ask for. - ▸IP whitelist. Add your own IP address to the list in your dashboard and connections from it need no password. Handy where there's nowhere to type one, such as Android's Wi-Fi proxy screen.
Which protocol?
A protocol is the language your device uses to talk to the proxy. Every proxy works over HTTP, HTTPS and SOCKS5; if an app asks for SOCKS5 by name, give it SOCKS5 details, and put HTTP details in an HTTP box.
Common slip-ups, and the quick fix for each
When something feels off on a first try, it's often one of these.
Logging in on a rotating address
Rotating means a new address for every request; sticky means one address kept for 1-30 minutes. Use sticky for anything you sign in to, or the site may log you out halfway through.
A slip in the proxy details
One wrong character in the host, port, username or password and nothing connects. Copy them again from your dashboard and try them in our proxy checker. Using the IP whitelist? Home IP addresses change now and then, so check yours with the IP checker and update the list.
Starting with a big run
Try one task on one proxy first and watch what happens. A mistake in a large run wastes data, and any accounts the tool uses can be limited.
A site keeps refusing the address
Some sites turn away addresses from server companies however gently you go. Retrying won't change that, so move just that site to residential.
Proxy not working when it arrives? Tell us within 24 hours of delivery and we'll swap it, or refund that part of the order if we can't (see the refund policy). Support answers 24/7 through the contact page.
Is a proxy for Scrapy allowed?
Using a proxy is legal in most countries (a few restrict them), so the question is what you do through it: stick to public information and your own accounts, at a sensible pace, within the service's terms and the law where you live.
Many sites and games ban bots and automation tools outright, so read the rules of whatever the tool touches before you start. Where automation is forbidden, running it through a proxy is still forbidden.
If personal data is involved, data-protection law applies as well, and no proxy can protect an account from a platform's own decisions.
Keeping the cost down
You pay for data by the GB, never for time, so set the proxy in the one browser or app that needs it and switch it off when you're done.
Start with a small order
Buy a little Datacenter data first, at $2.49 per GB, and try it where you need it. Top up once it works.
Use it within 30 days
Each order runs for 30 days from payment, and GB you haven't used don't carry over, so buy for the weeks ahead rather than the year.
Match the type to the site
Leave easy sites on datacenter and send only the stubborn ones through residential, mobile or premium. Mixing types keeps the bill small.
Delivered instantly · 99.9% uptime · no traffic logs · you pay the price on the live plan you choose
Questions beginners ask about Scrapy
Start small, then decide
Order a little Datacenter data, test it with our free tools, and buy more only once it does what you need.
Get Datacenter proxies