Track Amazon prices with Python: a small, polite scraper

In plain English: this script opens an Amazon product page and reads five things from it: the title, the price, the star rating, the number of reviews and whether the item is in stock. That's handy if you're waiting for something you want to buy to drop in price, or if you sell a few items and want to keep an eye on similar products. Amazon is one of the stricter sites about automated visits, though, so most of the work here is about going slowly and politely. Reading the page is the easy part. You'll use Python with requests to fetch pages and beautifulsoup4 to read them, and a proxy so the visits don't all come from your own IP address.

Quick start in 3 steps

1. Run the pip command from Step 1 to install everything.
2. Paste your proxy host, port and login into the dictionary in Step 2.
3. Run the script for one product code and check that a title and a price print out.

Is this allowed? The ground rules

Stick to product information that any visitor can see. Leave anything behind a login alone, along with personal details about the people who buy and sell there. Read Amazon's robots.txt (the file where a site tells automated programs what to leave alone) and its Terms of Service, and keep within them. When you need catalogue data on a larger scale, Amazon has official APIs for that, such as the Product Advertising API, and they're the proper, approved route. Don't try to beat a CAPTCHA or a sign-in page, keep your request rate low, and treat this script as a tool for light personal research, not for copying the catalogue. Making sure your use is permitted is your responsibility.

Which proxy type works on Amazon?

Amazon pays close attention to how many pages each IP address asks for, and it's quick to show a CAPTCHA or block an address that asks for too many too fast. Spreading your requests over several addresses keeps each one's share small. Amazon is especially wary of address ranges that belong to data centres, so residential proxies, which use household broadband addresses, are the dependable choice for product pages. If you still run into trouble on certain pages, residential premium gives you a little more room. Checking prices in another country's store? Use proxies in that country. Our proxy for scraping page explains how to match the proxy type to the job.

What you need

• Python 3, already installed, and a window for typing commands.
• The four proxy details: host, port, username and password.
• The product codes you want to follow. Amazon calls them ASINs, and Step 3 shows where to find one.

Step 1: Install the libraries

Type this into your terminal. Besides the two main libraries it installs lxml, a quick HTML reader that BeautifulSoup can use behind the scenes:

pip install requests beautifulsoup4 lxml

Step 2: Add your proxy details

Our proxies use the standard HOST:PORT layout with a username and password. Give them to requests as a dictionary, replacing the words in capitals with your own details:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

Step 3: Fetch a product page

Every Amazon product has an ASIN, a short code that appears in the page address as /dp/ASIN. The function below builds that address from the code, fetches the page through the proxy and returns its HTML:

import requests
from bs4 import BeautifulSoup

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

def fetch_product(asin, domain="www.amazon.com"):
    url = f"https://{domain}/dp/{asin}"
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=25)
    resp.raise_for_status()
    return resp.text

The headers tell Amazon what kind of browser is asking and which languages and page formats it accepts. timeout=25 stops the script waiting more than 25 seconds for an answer. The domain setting lets you point the same function at a different country's Amazon store later on.

Step 4: Read the five fields

Amazon's page layout varies between product categories and changes often. So for each field, the reader below tries a few known element names in turn and moves on quietly when one is missing. Write your own code the same way, assuming any single selector could come back empty:

def parse_product(html):
    soup = BeautifulSoup(html, "lxml")

    def first_text(*selectors):
        for sel in selectors:
            el = soup.select_one(sel)
            if el and el.get_text(strip=True):
                return el.get_text(strip=True)
        return None

    title = first_text("#productTitle")
    price = first_text(
        "span.a-price span.a-offscreen",
        "#priceblock_ourprice",
        "#priceblock_dealprice",
    )
    rating = first_text("span.a-icon-alt")
    review_count = first_text("#acrCustomerReviewText")
    availability = first_text("#availability span")

    return {
        "title": title,
        "price": price,
        "rating": rating,
        "review_count": review_count,
        "availability": availability,
    }

html = fetch_product("B08N5WRWNW")
print(parse_product(html))

When every field comes back as None (Python's word for "nothing here"), you've most likely been shown a robot-check page or a CAPTCHA instead of the product. Print the HTML and read it before you touch the selectors. The fix is nearly always to slow down and ask for fewer pages, not to rewrite the reader.

Step 5: Follow several products, slowly

For more than a couple of products, loop through them, take turns between proxy endpoints and pause for a random 4 to 10 seconds after each page:

import random
import time

PROXY_POOL = [
    "http://USER:PASS@HOST1:PORT",
    "http://USER:PASS@HOST2:PORT",
    "http://USER:PASS@HOST3:PORT",
]

def scrape_many(asins):
    out = {}
    for asin in asins:
        p = random.choice(PROXY_POOL)
        prox = {"http": p, "https": p}
        try:
            url = f"https://www.amazon.com/dp/{asin}"
            resp = requests.get(url, headers=headers, proxies=prox, timeout=25)
            resp.raise_for_status()
            out[asin] = parse_product(resp.text)
        except requests.RequestException as e:
            out[asin] = {"error": str(e)}
        time.sleep(random.uniform(4, 10))  # be a good citizen
    return out

If one product fails, the error is written down against that product and the loop moves on, so a single bad page can't stop the whole run. If your plan already changes the address on every request, one endpoint is enough, because the switching happens on our side. Whatever your plan, leave the random sleep in place. It's the politeness rule you don't get to skip.

Step 6: Save results and don't fetch twice

There's no need to fetch a page you already have. For price watching, save each result with the time it was collected and refresh on a schedule that suits you. Once a day is plenty for most people. Fewer visits are kinder to Amazon, and fewer visits also mean fewer blocks. This helper adds one line per check to a file called prices.jsonl, where each line is a small, self-contained record:

import json
from datetime import datetime

def save_snapshot(asin, data, path="prices.jsonl"):
    record = {"asin": asin, "fetched_at": datetime.utcnow().isoformat(), **data}
    with open(path, "a", encoding="utf-8") as f:
        f.write(json.dumps(record) + "\n")

Check before you run it

Confirm that your requests leave through the proxy: our my IP tool shows your own address, so you know what you're comparing against, and the proxy checker tests each endpoint in your list. It takes a couple of minutes and spares you a run that quietly used your home connection.

Common mistakes

• Checking prices every few minutes. Prices rarely change that fast, and frequent visits are what get addresses blocked. Daily is fine.
• Treating a robot-check page as a broken script. All-empty fields usually mean you went too fast, not that the code is wrong.
• Comparing different stores without noticing. Each country's Amazon store has its own prices and currency, so keep the domain and the proxy country matched.
• Deleting the pause "just for a test". Tests count as visits too.

Beginner questions

Where exactly do I find an ASIN?

Open the product in your browser and look at the address bar. The code comes right after /dp/ in the address. Copy that code into the script.

Can the script tell me when a price drops?

Not on its own, but the saved file makes it possible. Each line holds a product code, a time and a price, so a few extra lines of code can compare the newest price with the previous one for the same product.

Why is the price different from the one I see in my browser?

Your browser may be signed in, set to another delivery address or currency, or showing a deal tied to your account. The script sees the page as a visitor in the proxy's country does.

A good Amazon scraper is a patient one. It asks slowly, saves what it finds and reads pages carefully. Stay with public product pages, follow Amazon's terms, and move to the official Product Advertising API if your project outgrows light personal research.