Scrape Google search results with Python: a starter guide

In plain English: web scraping means writing a short program that opens web pages and copies out the bits you need, so you don't have to do it by hand. Here the page is a Google results page and the bits are each result's title and link. A typical first use is checking where your own blog or shop appears for a few searches. You'll write it in Python with two add-on libraries: requests, which fetches pages, and beautifulsoup4, which reads them. A proxy sends the searches from a different IP address, so they don't all come from your home connection, and it lets you see results the way people in another country see them. One honest warning before you start: Google limits automated searching, so this script is for a few polite queries, not thousands.

Quick start in 3 steps

1. Install the two libraries with the one-line command in Step 1.
2. Copy your proxy host, port, username and password into the proxies settings shown in Step 2.
3. Run a single search, look at the titles and links it prints, and only then think about adding more searches, with pauses in between.

Is this allowed? Read this before any code

Set yourself some rules first. Collect only what anyone can see without signing in. Read Google's robots.txt (a public file in which a site says which areas automated programs should keep out of) and its Terms of Service, and stay inside them. Never try to get past a login or a CAPTCHA, the "prove you're human" puzzle. Leave plenty of time between requests so the service stays usable for everyone else. Tracking a few rankings for your own SEO research is a common reason to do this, but checking that your use is permitted is down to you, and an official API is the better choice whenever one covers what you need. Our proxy for scraping and proxy for SEO pages say more about this.

Why a proxy, and which type?

Google links the searches it receives to the IP address they arrive from. Send a string of searches from one address and you'll soon be slowed down or blocked for a while, even at a gentle pace. Sharing the searches between several addresses keeps the number coming from each one low. For the type, start with residential proxies: their addresses are ones that internet providers hand out to homes, and they usually get on better with search pages than server addresses do. If standard residential still struggles, residential premium is the step up. Care about rankings in one particular country? Pick proxies in that country, because Google shows different results in different places.

What you need

• Python 3 on your computer, plus a terminal to type commands into (Command Prompt or PowerShell on Windows, Terminal on a Mac).
• Your proxy details: the host, port, username and password from your dashboard.
• A short list of searches, ideally ones you'd happily type in yourself.

Step 1: Install the libraries

A library is ready-made code you can reuse. Install both with pip, the installer that comes with Python, by typing this into your terminal:

pip install requests beautifulsoup4

Step 2: Add your proxy details

Our proxy details follow the familiar HOST:PORT pattern, plus a username and password. The requests library expects them in a small dictionary (a list of named values) like the one below. Swap in your own details:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

Why two lines? requests picks the proxy by the kind of address it's fetching: one entry for http:// pages, one for https:// pages. Both point at the same proxy.

Step 3: Fetch one results page

This function turns your search words into a Google address, fetches it through the proxy and returns the page's HTML (the code a browser turns into the page you see):

import requests
from bs4 import BeautifulSoup
from urllib.parse import urlencode

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch_results(query, lang="en"):
    params = {"q": query, "hl": lang}
    url = "https://www.google.com/search?" + urlencode(params)
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=20)
    resp.raise_for_status()
    return resp.text

• headers are extra lines of information sent with the request. User-Agent names the browser and system making the request, and Accept-Language asks for English.
• "hl": lang adds the interface language to the address, English by default.
• timeout=20 means the script gives up with an error if no answer arrives within 20 seconds, instead of waiting forever.
• raise_for_status() turns an error reply, a block for example, into a clear Python error rather than letting the script carry on with a bad page.

Step 4: Pick out the titles and links

Google reshuffles the layout of its results often, so the part that reads the page (the parser) should lean on simple, sturdy clues rather than long class names that change. This one goes through each result block, takes the first link that has an href (the address a link points to) and the h3 heading inside the same block. Expect to adjust the selectors when the layout moves:

def parse_results(html):
    soup = BeautifulSoup(html, "html.parser")
    results = []
    for block in soup.select("div.g"):
        link = block.find("a", href=True)
        title = block.find("h3")
        if not link or not title:
            continue
        results.append({
            "title": title.get_text(strip=True),
            "url": link["href"],
        })
    return results

html = fetch_results("best running shoes")
for i, r in enumerate(parse_results(html), start=1):
    print(f"{i}. {r['title']} -> {r['url']}")

An empty list rarely means Google found nothing. Usually the layout has changed, or the reply was a different page altogether, such as a consent screen or a robot check. Print html[:2000], the first 2000 characters of what came back, and read it before you start changing selectors.

Step 5: Space your searches out

Once you go past a handful of searches, take turns between several proxy endpoints and wait a random few seconds between requests. In the helper below, a different endpoint is picked for each search, and the script then pauses for somewhere between 3 and 8 seconds:

import random
import time

PROXY_POOL = [
    "http://USER:PASS@HOST1:PORT",
    "http://USER:PASS@HOST2:PORT",
    "http://USER:PASS@HOST3:PORT",
]

def pick_proxy():
    p = random.choice(PROXY_POOL)
    return {"http": p, "https": p}

def polite_fetch(query):
    proxies = pick_proxy()
    url = "https://www.google.com/search?" + urlencode({"q": query, "hl": "en"})
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=20)
    resp.raise_for_status()
    time.sleep(random.uniform(3, 8))  # pace yourself
    return resp.text

Set a residential plan to rotating in your dashboard and the address changes for you as you go, so one endpoint is enough. Keep the random pause either way. No other habit here matters as much: it keeps you polite, and it's also what stops you being slowed down.

Step 6: Retry gently, then give up

Connections fail from time to time. The function below tries up to three times and waits a bit longer after each failure: 1 second, then 2, then 4. Programmers call that pattern exponential backoff.

def fetch_with_retry(query, attempts=3):
    for attempt in range(attempts):
        try:
            return polite_fetch(query)
        except requests.RequestException as e:
            wait = 2 ** attempt
            print(f"Attempt {attempt + 1} failed: {e}; waiting {wait}s")
            time.sleep(wait)
    return None

When attempts keep failing, read that as a cue to slow right down or call it a day, not to push harder. Repeated blocks nearly always mean you've asked for too much, too quickly.

Check your setup before a real run

Make sure the script's traffic really leaves through the proxy. Note your own address on our my IP tool, so you can tell it apart from the proxy's, and test every endpoint on your list with the proxy checker. Two minutes of checking beats discovering afterwards that a whole run went out on your home connection.

Common mistakes

• Removing the pause to save time. Without it, blocks arrive quickly and the run ends early anyway.
• Believing an empty list. It's usually a changed layout or a check page, not a search with no results.
• Filling in only the "http" line. Google's addresses start with https://, so without the "https" entry your searches go out over your own connection.
• Answering a block with more force. Extra retries or extra addresses don't make heavy scraping acceptable. Slow down, or move to an official API.

Questions beginners ask

Why does the script get a different page from my browser?

Your browser carries cookies, saved settings and a location that the script doesn't have, so Google may show it a different version, a consent page or a check page. That's normal. Print the start of the HTML to see which one you received.

How many searches is reasonable?

Think in handfuls rather than thousands: a few searches with long pauses, run now and again. If your project needs far more than that, an official API is the right tool for it.

Does the proxy need to be in a particular country?

Only when local rankings matter to you. Results differ between countries, so choose a proxy in the country you're checking. With residential plans, you pick the country in your dashboard after you buy.

Start small: one query, a long pause and a careful look at what came back. Respect the terms of every service you query, and switch to an official API as soon as one fits what you're doing.