Python and Rotating Proxies: A Gentle First Walkthrough
Rotating proxies simply means sending your requests out through different IP addresses instead of always the same one. In Python it's easier than it sounds, and there are two tidy ways to do it. The simplest hands the job to a rotating gateway: you point the requests library, a widely used tool for fetching web pages, at it, and the pool switches IPs for you. The other is to keep your own list of proxy addresses and cycle through it. You'll find both below, with working examples and a sensible pace.
Every example is meant for legitimate collection of public data. Keep to each site's terms of service, its robots.txt rules for automated visitors and the law. And since no proxy can promise access, write your code to cope calmly with errors.
Written and kept up to date by the people who run Budget Proxies. If a step no longer matches what you see on your screen, let us know through the contact page.
What this guide walks you through
- The easy way: let a gateway swap IPs for you
- The hands-on way: step through a list of your own
- Staying on one IP for a short task, with pauses
- Tips that save beginners trouble
- Getting your script connected
The easy way: let a gateway swap IPs for you
Here you only ever talk to one address, the gateway. Behind it, the pool picks a new IP for each request, so there's nothing for you to keep track of. All requests needs is a proxies dict (a small dictionary telling it which proxy to use):
import requests
# One rotating gateway; the pool changes the exit IP for you.
proxies = {
"http": "http://USER:[email protected]:8000",
"https": "http://USER:[email protected]:8000",
}
resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=20)
print(resp.json()) # a different exit IP on subsequent calls
This is the approach we suggest for beginners: we look after the rotation, the location targeting and the size of the pool, and your code stays short.
The hands-on way: step through a list of your own
If you have a list of separate proxy addresses (for example static IPs that don't change), you can rotate through them yourself. Python's itertools.cycle goes round the list in order and starts again at the top when it reaches the end:
import requests
from itertools import cycle
proxy_list = [
"http://USER:[email protected]:8000",
"http://USER:[email protected]:8000",
"http://USER:[email protected]:8000",
]
pool = cycle(proxy_list)
urls = ["https://httpbin.org/ip"] * 6
for url in urls:
proxy = next(pool)
proxies = {"http": proxy, "https": proxy}
try:
r = requests.get(url, proxies=proxies, timeout=20)
print(r.status_code, r.json())
except requests.RequestException as e:
print("request failed:", e) # skip and continue
Staying on one IP for a short task, with pauses
A requests.Session reuses one connection across several steps, which helps when a task has to happen in order. Add a short, random pause between requests to keep your pace reasonable:
import requests, time, random
session = requests.Session()
session.proxies = {
"http": "http://USER:[email protected]:8000",
"https": "http://USER:[email protected]:8000",
}
for path in ["/", "/page/2", "/page/3"]:
r = session.get("https://example.com" + path, timeout=20)
print(r.status_code)
time.sleep(random.uniform(1.0, 3.0)) # be polite
The table below shows when to use each approach.
| Approach | Good for | Effort |
|---|---|---|
| Let the gateway rotate | Most scraping; the pool picks the IPs | Very little (one proxies dict) |
| Go round your own list | Fixed or static IPs you look after | A little (itertools.cycle) |
| One session, with pauses | Tasks with several steps in order | A little (Session plus a pause) |
Tips that save beginners trouble
- Give every request a timeout, and put it inside try/except (Python's way of catching errors); then a single failure can't bring the whole script down.
- Retry temporary errors (timeouts, and server errors starting with a 5) with exponential backoff, waiting longer between each try, rather than giving up at the first failure.
- Slow down with random pauses and only a few requests at once, so the load on the site stays reasonable.
- Choose the right pool for each site: datacenter (server IPs) is fine where protection is light; use residential (home IPs) where it's strict or the content depends on location.
Getting your script connected
Every plan supports HTTP/HTTPS and SOCKS5 (for SOCKS, install requests[socks]), with a username/password login or an IP whitelist that approves your own address, across 55+ countries. Sizing help is on the proxy calculator, the rates are on the pricing page, and you can buy now when you're ready. Want the whole process from start to finish? Our beginner's walkthrough, how to scrape a website, covers it.
Questions beginners ask
Not sure what to buy? Start at step 1
Here are our four types, from the lowest price per GB upwards. Try a small amount of the first one that could work for your project, and only move up a step if a quick test lets you down.
Sites without bot protection, big batches of public pages, automation and SEO tools
Protected sites, ad checks, market research and seeing what one place really sees
Apps and platforms built for phones, plus testing your own mobile app
Well-guarded sites, where a better success rate is worth the higher price
Small first order? Datacenter from $2.49/GB · billed on the data you actually move · ready straight away · 30 days to use it · no traffic logs.
What to read next
What Are Proxies? A Starting Point for Beginners
A proxy passes your traffic on, so websites see its address rather than yours. How it works, the types you ca…
Read the guide →What Are Datacenter Proxies? The Low-Cost Place to Start
Datacenter proxies use addresses from server companies, so they're quick and low-cost. How they work, where t…
Read the guide →What Are Residential Proxies? Home Addresses, Explained
A residential proxy sends your requests through a real home connection, so sites see a local visitor. How it …
Read the guide →Datacenter vs Residential Proxies: Which Should a Beginner Buy?
Datacenter is quick and low-cost; residential is trusted more. A plain comparison of speed, price and success…
Read the guide →Mobile vs Residential Proxies: Do You Really Need Mobile?
Mobile proxies use phone-network IPs; residential ones use home broadband. How trust, cost and uses differ, a…
Read the guide →Proxy or VPN? The Difference, and Which One You Need
Already have a VPN app? How a proxy differs, what each one is good for, and why a proxy suits jobs like check…
Read the guide →Got a particular job in mind?
Ready to try your first proxy?
Choose from 55+ countries, pay from $2.49/GB for only the data you use, and get your login details straight away. If a step trips you up, human support is there 24/7.
