Python requests through a proxy: a starter guide

In plain English: requests is a popular Python library for fetching web pages and talking to APIs (services that answer with data rather than web pages). To make it use a proxy, you pass it a small dictionary: a set of labelled values saying which proxy handles which kind of traffic. That really is the whole trick. The rest of this guide adds the habits that keep a small script running smoothly: sessions, rotation, retries and SOCKS5.

Quick start in 3 steps

1. Install the library with pip.
2. Copy the dictionary example below and add your proxy details.
3. Run it. If the IP it prints belongs to the proxy, you're connected.

Have these to hand

• Python 3 and a terminal.
• Your proxy details. The examples write them as HOST, PORT, USER and PASS.
• The requests library, which one command installs:

pip install requests

The proxy dictionary

The keys (the labels on the left) tell requests which traffic to send through the proxy, and the values are the proxy's address with your login included. Usually both point at the same proxy:

import requests

proxies = {
    "http": "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=20)
print(r.json())

Look closely and you'll notice something that puzzles nearly everyone at first: the https key has a value starting with http://, not https://. That's correct. The value describes how your script talks to the proxy. For secure sites, the proxy opens a tunnel using a method called HTTP CONNECT, and your encrypted traffic passes through it untouched. If the printed IP matches the proxy's, you're in.

Passwords with awkward characters

If your username or password contains @, : or /, Python reads those as part of the address and gets muddled. Encode them first, which turns each awkward character into a safe code:

from urllib.parse import quote

user = quote("USER")
pwd = quote("P@SS/word")
proxy = f"http://{user}:{pwd}@HOST:PORT"
proxies = {"http": proxy, "https": proxy}

Use a Session for several pages

A Session keeps one connection open between requests instead of starting from scratch each time, and lets you set the proxy, headers and other options once:

import requests

session = requests.Session()
session.proxies.update({
    "http": "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
})
session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"})

for path in ["/", "/about", "/contact"]:
    resp = session.get(f"https://example.com{path}", timeout=20)
    print(path, resp.status_code)

Reusing the connection makes multi-page jobs quicker, and your code tidier.

Taking turns across several endpoints

If you have a list of sticky endpoints (addresses that hold on to the same IP), cycle through them so each request uses the next one in line:

import requests, itertools

endpoints = [
    "http://USER:PASS@HOST:PORT1",
    "http://USER:PASS@HOST:PORT2",
    "http://USER:PASS@HOST:PORT3",
]
pool = itertools.cycle(endpoints)

def fetch(url):
    proxy = next(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

print(fetch("https://api.ipify.org?format=json").json())

With a single rotating gateway you can skip this: keep one dictionary, and each request leaves from a different IP automatically.

Retry politely when something fails

Now and then a request will time out or hit an IP having a bad moment. Rather than writing your own loop, attach a retry adapter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(total=4, backoff_factor=1.5,
              status_forcelist=[429, 500, 502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))
session.proxies.update({"http": "http://USER:PASS@HOST:PORT",
                        "https": "http://USER:PASS@HOST:PORT"})

A 429 status is the site asking you to slow down. The backoff_factor makes each retry wait longer than the one before, giving the server room to breathe instead of hammering it.

Switching to SOCKS5

Requests needs one extra package to speak SOCKS5:

pip install "requests[socks]"

Then change the start of each address:

proxies = {
    "http": "socks5h://USER:PASS@HOST:PORT",
    "https": "socks5h://USER:PASS@HOST:PORT",
}

The h in socks5h asks the proxy to look up website names itself, so those look-ups (the DNS step) don't leak out through your own connection. For scraping, it's nearly always the setting you want.

Check it worked

The ipify examples above print the address a website sees, so that's your first check. For a second opinion, put the same details into our proxy checker.

Which type should I pick?

• Practice, APIs and easy-going sites: datacenter proxies, quick and lower in cost per GB.
• Sites that refuse data-centre traffic: residential proxies first, then residential premium if they still say no.
• Services that expect phone-network visitors: mobile proxies.

Working on search rankings? Our notes on a proxy for SEO cover that. General scraping advice lives in the proxy for scraping guide, and every price is on the pricing page.

Common mistakes

• https:// in the proxy value. For an ordinary HTTP proxy, both values start with http://.
• No timeout. Without one, a request through an unresponsive proxy can hang for a very long time. The examples use timeout=20, which means 20 seconds.
• Unencoded special characters. Run awkward usernames and passwords through quote() as shown.
• Requests fired back to back. Add a short pause between them.

Is this allowed?

Before you scrape anything, read the site's robots.txt, follow its terms of service, wait a moment between requests with a short time.sleep(), respect 429 responses with a retry policy like the one above, and collect only public data you're permitted to use. Don't use proxies to get past logins, defeat anti-fraud systems or commit fraud.

Beginner questions

Why are there two keys, http and https?

Each key covers one kind of website: http for plain addresses, https for secure ones. Set both, or some of your requests will skip the proxy.

How do I see which IP a request used?

Fetch api.ipify.org through the same proxy, as the first example does. It replies with the address it saw.

Do I need a Session?

Not for one or two requests. Once you're fetching several pages from the same site, a Session saves time.