BeautifulSoup for beginners: reading HTML with Python

In plain English: every web page is built from HTML, plain text with tags such as <h1> for a heading or <a> for a link wrapped around the content. BeautifulSoup is a Python library that reads that HTML and lets you ask simple questions of it, like "give me every link" or "what does the price box say?". It copes well with the untidy HTML found on real websites, which is a big part of why beginners like it. Here you'll start with a small practice page on your own computer, learn the handful of commands that do most of the work, and then fetch live pages with requests, optionally through a proxy.

Quick start in 3 steps

1. Run the one-line install from Step 1.
2. Copy the practice example from Step 2 into a file, run it, and check that it prints "Welcome" and "/more".
3. Try the search commands from Steps 3 and 4 on the same practice page before you go anywhere near a live site.

Is this allowed?

Reading HTML that's already on your computer harms nobody. Once you start downloading pages from a live website, though, the usual rules apply. Read the site's robots.txt (its public list of areas that automated programs should avoid) and its Terms of Service, take only public data you're permitted to use, space out your requests so you don't overload anything, and never work around logins or anti-bot checks. Good manners are what keep your scripts working and the websites you rely on healthy.

What you need

• Python 3, plus a terminal for the install command.
• A text editor to save your scripts in. Any plain one will do.
• Proxy details, but only for Step 6. Everything before that runs offline.

Step 1: Install BeautifulSoup

You need the library itself and a parser, the engine that actually reads the HTML. Python's built-in html.parser works, but lxml is quicker and deals more gracefully with broken pages. This command installs BeautifulSoup, lxml and requests in one go:

pip install beautifulsoup4 lxml requests

Step 2: Your first parse

Give BeautifulSoup some HTML and the name of a parser, and it builds a "soup", a version of the page that Python can search. After that, you can step into the page with dots, the way you'd follow folders on a computer:

from bs4 import BeautifulSoup

html = """
<html>
  <body>
    <h1 class="title">Welcome</h1>
    <p class="intro">Hello <a href="/more">world</a>.</p>
  </body>
</html>
"""

soup = BeautifulSoup(html, "lxml")
print(soup.h1.text)            # Welcome
print(soup.p.a["href"])        # /more
print(soup.title)              # None (no title tag here)

The last line prints None, Python's way of saying "nothing found", because the practice page has no title tag. Remember that word. You'll meet it often.

Step 3: Finding things with find and find_all

Two commands do most of the searching. find gives you the first match, and find_all gives you a list of every match. You can search by tag name, by attributes (the extra details inside a tag, such as its class) or by both together:

# first paragraph with class "intro"
intro = soup.find("p", class_="intro")

# every link on the page
for link in soup.find_all("a"):
    print(link.get("href"), "->", link.text)

# filter by any attribute
button = soup.find("button", attrs={"data-role": "submit"})

Notice the underscore in class_. Python already uses class as one of its own words, so BeautifulSoup asks you to write class_ instead.

Step 4: CSS selectors with select

If you've ever written CSS, the language that styles web pages, select and select_one will feel familiar, and they're often the quickest way in. They accept the same selectors you'd write in a stylesheet. In the example, #results means "the element with the id results" and li.product means "list items with the class product":

# all items inside a list with id "results"
for item in soup.select("#results li.product"):
    name = item.select_one(".name").get_text(strip=True)
    price = item.select_one(".price").get_text(strip=True)
    print(name, price)

Adding strip=True to get_text trims the spaces and line breaks around the text, which saves a lot of cleaning up later.

Step 5: Moving around the page

Sometimes the value you want has no handy label of its own, but it sits next to, inside or above something that does. HTML works a bit like a family tree, and BeautifulSoup lets you move along it:

row = soup.find("td", string="Price")
value = row.find_next_sibling("td").text   # the cell after "Price"

parent_div = row.parent                     # go up one level
children = list(parent_div.children)        # immediate children

The helpers to know are find_next_sibling and find_previous_sibling (the element just after or just before, on the same level), parent (one level up), children (everything one level down) and find_parent (search upwards). They come in handy whenever an element has no convenient class to aim at.

Step 6: A live page, through a proxy

Practice pages are great for learning, but real projects mean fetching pages from the web. Here requests downloads the page and BeautifulSoup reads it. A proxy is optional for a single page. It becomes useful when you fetch many pages from one site, or when you want to see a page the way visitors in another country do. Datacenter proxies are quick and kind to a small budget on straightforward sites. Residential proxies, which use household broadband addresses, are the better match for sites that turn away server addresses. You'll find all four types on the proxies page.

import requests
from bs4 import BeautifulSoup

proxies = {
    "http": "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}
headers = {"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"}

resp = requests.get(
    "https://example.com/products",
    headers=headers,
    proxies=proxies,
    timeout=20,
)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "lxml")
for card in soup.select("div.product-card"):
    title = card.select_one("h2")
    price = card.select_one(".price")
    if title and price:
        print(title.get_text(strip=True), price.get_text(strip=True))

Look at the User-Agent in that example. It openly names your script, MyScraper, which is an honest way to introduce yourself to a website. timeout=20 stops the script waiting more than 20 seconds, and raise_for_status() stops it with a clear error if the site replies with one.

Real pages are messy: four good habits

• Always check that find or select_one actually found something before you read its text. A missing element comes back as None, and asking None for its text crashes the script.
• Use get("href") rather than ["href"] when an attribute might be missing. get quietly returns None, while the square-bracket version raises an error.
• Prefer lxml. It's faster and it copes with tags that were never closed.
• When many elements share a class name, start your selector from a parent with a unique id or data attribute so you only catch the ones you mean.

Check your setup

Before any real run, make sure the proxy is alive with our proxy checker, and confirm your own address with the my IP tool so you can tell the two apart. A couple of minutes now saves a confusing hour of debugging later.

Common mistakes

• Writing class instead of class_. Python reads it as its own keyword and stops with an error.
• Using select when you wanted one result. select always returns a list, even if it holds only one item. Use select_one for a single element.
• Reading the page before checking the download worked. An error page has HTML too, so check the reply first.
• Blaming BeautifulSoup for empty results on JavaScript-heavy sites. It can only read the HTML it receives.

Beginner questions

Should I use html.parser or lxml?

Either works for learning. html.parser needs no extra install, and lxml is faster and more forgiving with broken pages. Once lxml is installed, it makes a sensible default.

Can BeautifulSoup run JavaScript?

No. It reads the HTML it's given and nothing more. If a site builds its content with JavaScript after the page loads, you need a tool that drives a real browser, such as Playwright or Selenium, and then you can hand that browser's HTML to BeautifulSoup.

Do I need a proxy to use BeautifulSoup?

No. BeautifulSoup only reads HTML. A proxy matters only for the downloading step, and only when you fetch a lot of pages or need a particular location.

A little practice goes a long way. Once find, select and moving around the page feel natural, most pages become routine to read. Pair those skills with polite request habits and a dependable proxy, and you have a solid footing for any small data project.