Selenium with a proxy: a beginner-friendly Python guide

In plain English: Selenium is a Python library that opens a real browser, such as Chrome or Firefox, and clicks, scrolls and types for you. People use it to test their own websites and to read pages that only fill in after JavaScript has run. If your proxy doesn't need a password, adding it takes a single line. If it does, the browser pops up a sign-in box that Selenium can't fill in, and that box is where most first attempts get stuck. This guide covers both cases.

Quick start in 3 steps

1. Test your proxy details with one cURL command, so you know they work before Selenium is involved.
2. Install Selenium with pip, Python's package installer.
3. Pick the method that matches your proxy: Method 1 if you use IP whitelisting, Method 2 or 3 if you sign in with a username and password.

Have these to hand

• Python with pip, and a terminal to type commands into.
• Chrome or Firefox. Selenium 4.6 and later fetches the matching driver by itself through a feature called Selenium Manager. (The driver is a small helper program that lets Selenium steer the browser.) So there's no chromedriver to download by hand.
• Your four proxy details. In the code they appear as HOST, PORT, USER and PASS; swap in your own.

First, test the proxy on its own

Run this in your terminal. It asks a simple service which IP address it can see:

curl --proxy http://USER:PASS@HOST:PORT https://api.ipify.org

If the answer is the proxy's IP rather than yours (our My IP tool shows your own for comparison), you're ready. If it hangs or prints an error, sort the details out now. Otherwise you could spend an evening hunting a Selenium bug that turns out to be a typo.

Install Selenium

pip install selenium

Method 1: no password needed (IP whitelisting)

When your IP address is on the approved list in your dashboard, the proxy doesn't ask for a login, and one launch argument is enough:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--proxy-server=http://HOST:PORT")

driver = webdriver.Chrome(options=options)
driver.get("https://api.ipify.org")
print(driver.page_source)
driver.quit()

Trouble starts when a login is involved. Put USER:PASS@ in front of the host and Chrome ignores it, then shows its own sign-in box, which Selenium can't click. Methods 2 and 3 solve that.

Method 2: Chrome with a login, via a tiny extension

An extension is a small add-on for the browser. This script writes one on the spot, with your proxy details inside, and loads it into Chrome. The extension answers the proxy's login request through Chrome's webRequest feature, so the sign-in box never appears:

import zipfile, tempfile, os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

HOST, PORT, USER, PASS = "HOST", "PORT", "USER", "PASS"

manifest = """
{
  "version": "1.0.0",
  "manifest_version": 3,
  "name": "Proxy Auth",
  "permissions": ["proxy", "webRequest", "webRequestAuthProvider"],
  "host_permissions": ["<all_urls>"],
  "background": { "service_worker": "background.js" }
}
"""

background = f"""
chrome.runtime.onInstalled.addListener(() => {{
  chrome.proxy.settings.set({{
    value: {{
      mode: "fixed_servers",
      rules: {{ singleProxy: {{ scheme: "http", host: "{HOST}", port: {PORT} }} }}
    }},
    scope: "regular"
  }});
}});
chrome.webRequest.onAuthRequired.addListener(
  () => ({{ authCredentials: {{ username: "{USER}", password: "{PASS}" }} }}),
  {{ urls: ["<all_urls>"] }},
  ["blocking"]
);
"""

path = os.path.join(tempfile.mkdtemp(), "proxy_auth.zip")
with zipfile.ZipFile(path, "w") as zf:
    zf.writestr("manifest.json", manifest)
    zf.writestr("background.js", background)

options = Options()
options.add_extension(path)

driver = webdriver.Chrome(options=options)
driver.get("https://api.ipify.org")
print(driver.page_source)
driver.quit()

The login keeps working as the browser moves from page to page, and it works with a visible window. To run headless (with no window at all), use the newer --headless=new flag, because the older headless mode doesn't load extensions.

Method 3: Firefox, or Chrome, with selenium-wire

Firefox handles proxies through its profile settings rather than extensions, and an easy way to give it a username and password from Python is the selenium-wire package. Install it first:

pip install selenium-wire blinker==1.7.0

Then run:

from seleniumwire import webdriver

options = {
    "proxy": {
        "http": "http://USER:PASS@HOST:PORT",
        "https": "http://USER:PASS@HOST:PORT",
        "no_proxy": "localhost,127.0.0.1",
    }
}

driver = webdriver.Firefox(seleniumwire_options=options)
driver.get("https://api.ipify.org")
print(driver.page_source)
driver.quit()

The same block drives Chrome too, if you change Firefox to Chrome in the driver line, which makes it a shortcut for anyone who'd rather skip Method 2. The install command holds blinker at an older version on purpose: newer releases removed something selenium-wire still depends on.

Check the IP, and rotate if you need to

Each example prints the page from api.ipify.org, and that page holds the IP address the website saw. Make sure it's the proxy's, not yours. You can also run the same details through our proxy checker. Planning lots of visits? A rotating endpoint, which hands out a fresh IP address for each new session, gives every browser you start its own address and spreads your visits out.

Which type should I pick?

A real browser downloads everything on a page: pictures, fonts and scripts. That uses more data than a plain request, so keep your test runs small while you're learning.

• Practice runs, simple sites and your own projects: datacenter proxies keep practice cheap.
• Sites that turn away data-centre addresses: residential proxies, or residential premium for the fussiest ones. Both use addresses from home internet connections.
• Mobile-app content or pages tied to a phone network: mobile proxies.

Current rates for all four are on the pricing page.

Common mistakes

• Login details in the launch argument. Chrome ignores them and shows the sign-in box. Use Method 2 or 3.
• The old headless flag with Method 2. The extension won't load. Switch to the new headless mode.
• Letting blinker update. If selenium-wire breaks after an upgrade, check that blinker is still on the version from the install command.
• Skipping driver.quit(). Every example ends with it. Leave it out and forgotten browsers pile up in the background, using memory.

Is this allowed?

Selenium makes it easy to visit a site far more often than a person ever would. Before you scale up, read the site's robots.txt file (a plain text file listing the parts of the site automated tools should leave alone), follow its terms of service, leave deliberate pauses between actions, and collect only public information you're allowed to use. Never use automation to get past a login or a paywall, or to create fake accounts. A slow, polite script that spreads its visits sensibly keeps working for far longer than a pushy one. Our guide to choosing a proxy for scraping goes into more detail.

Beginner questions

Which method should I start with?

If you can whitelist your IP, Method 1 is the shortest. If you need a username and password, Method 3 has the least code, and Method 2 avoids installing extra packages.

Can I watch the browser while it works?

Yes. The examples don't use a headless flag, so a normal window opens and you can follow every step.

Do I still need to download chromedriver?

Not with Selenium 4.6 or newer. Selenium Manager fetches the right driver the first time your script runs.