Getting YouTube data with Python: API first, browser second

In plain English: YouTube has an official API, a doorway built for programs, which hands back tidy facts about videos and channels: titles, view counts, likes and comment counts. You don't need to read any web pages to use it. Each project gets a daily quota, an allowance of requests, so you plan around that. Use the API first. For the few things it doesn't cover, such as how a page looks to viewers in a particular country, you can have Python drive a real Chrome browser with a tool called Selenium, and a proxy decides which country YouTube thinks you're in. This guide covers both, in that order.

Quick start in 3 steps

1. Create an API key in the Google Cloud Console and switch on "YouTube Data API v3" for your project.
2. Install Google's Python client with the command in Part 1.
3. Paste your key into the video example, run it with one video ID and read the numbers it prints.

Is this allowed? Start with YouTube's rules

YouTube's robots.txt file and its Terms of Service set out what's allowed, and the official API is the approved way to read most of its data. Keep to public, non-personal information, stay inside the API's daily quota, and never reach for anything behind a login or try to defeat YouTube's anti-bot checks. Save your results so you don't repeat calls for no reason, and keep your pace steady. Following these rules is more than good manners. It's what keeps your access working month after month.

What you need

• Python 3, used from your usual terminal.
• A Google account, which you'll use to sign in to the Google Cloud Console and create the API key.
• Google Chrome, but only for the browser part.
• Proxy details, also only for the browser part.

Part 1: The official YouTube Data API

For facts and figures like video statistics and channel details, the API is more dependable than reading web pages, every time. Once your key is ready, install the client library:

pip install google-api-python-client

Then ask for a video's statistics using its ID. The ID is the short code in a video's address: the part after v= in a normal link, or after youtu.be/ in a short one.

from googleapiclient.discovery import build

API_KEY = "YOUR_API_KEY"
youtube = build("youtube", "v3", developerKey=API_KEY)

def get_video_stats(video_id):
    request = youtube.videos().list(
        part="snippet,statistics",
        id=video_id,
    )
    response = request.execute()
    item = response["items"][0]
    return {
        "title": item["snippet"]["title"],
        "channel": item["snippet"]["channelTitle"],
        "views": item["statistics"].get("viewCount"),
        "likes": item["statistics"].get("likeCount"),
        "comments": item["statistics"].get("commentCount"),
    }

print(get_video_stats("dQw4w9WgXcQ"))

The figures are read with .get, so if YouTube leaves one out, you get None instead of a crash.

Searching, one page at a time

Search results come back in pages of up to fifty videos. Each page includes a nextPageToken, a bookmark you send back to get the following page. The loop below follows those bookmarks and stops when it has enough or when no bookmark is returned:

def search_videos(query, max_pages=2):
    results = []
    page_token = None
    for _ in range(max_pages):
        resp = youtube.search().list(
            part="snippet",
            q=query,
            type="video",
            maxResults=50,
            pageToken=page_token,
        ).execute()
        for item in resp["items"]:
            results.append({
                "video_id": item["id"]["videoId"],
                "title": item["snippet"]["title"],
            })
        page_token = resp.get("nextPageToken")
        if not page_token:
            break
    return results

Searching uses up your daily quota much faster than looking up single videos, so keep max_pages small while you experiment.

Part 2: When you need a real browser

A few things simply aren't in the API, such as how a page appears to viewers in one particular region. YouTube builds its pages with JavaScript, code that runs inside the browser after the page arrives, so an ordinary download returns very little you can use. Selenium solves that by controlling a real Chrome browser, which runs the JavaScript for you. Install it, and make sure you have an up-to-date Chrome and the matching driver:

pip install selenium

Which proxy type should the browser use?

YouTube adjusts what it shows to your location and keeps an eye out for automated traffic, so one address driving page after page will soon be shown CAPTCHAs. Residential proxies give each browser session a household broadband address in the country you choose, and residential premium adds steadiness over long sessions. When the results you're studying depend on region, a proxy in that country is a must, not an extra. All the options are on the proxies page.

Giving Selenium a proxy with a login

Chrome's command-line proxy option won't accept a username and password, so this guide uses the selenium-wire package, which handles proxies that need a login:

pip install selenium-wire

from seleniumwire import webdriver
from selenium.webdriver.common.by import By
import time

seleniumwire_options = {
    "proxy": {
        "http": "http://USER:PASS@HOST:PORT",
        "https": "http://USER:PASS@HOST:PORT",
        "no_proxy": "localhost,127.0.0.1",
    }
}

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--lang=en-US")

driver = webdriver.Chrome(
    seleniumwire_options=seleniumwire_options,
    options=options,
)

Three settings are worth understanding. --headless=new runs Chrome without opening a window. --lang=en-US asks for English pages. no_proxy keeps traffic to your own computer off the proxy. One honest note: selenium-wire is a third-party package, so check its project page for its current status and which Selenium versions it supports before you rely on it.

Reading a video page

With the browser running, open a video's page, give the JavaScript a few seconds to fill it in, then read elements by their selectors. YouTube changes its page structure from time to time, so each read is wrapped in a try-and-catch: if an element is missing, you get None and the script carries on.

def scrape_watch_page(video_id):
    driver.get(f"https://www.youtube.com/watch?v={video_id}")
    time.sleep(5)  # let dynamic content load

    try:
        title = driver.find_element(
            By.CSS_SELECTOR, "h1.ytd-watch-metadata"
        ).text
    except Exception:
        title = None

    try:
        channel = driver.find_element(
            By.CSS_SELECTOR, "ytd-channel-name #text a"
        ).text
    except Exception:
        channel = None

    return {"video_id": video_id, "title": title, "channel": channel}

data = scrape_watch_page("dQw4w9WgXcQ")
print(data)
driver.quit()

Habits that keep things reliable

• Use the API for anything it offers. It's quicker, tidier and comes with a quota you can plan around.
• In Selenium, wait for pages to finish loading rather than relying on one fixed pause. Pages take different amounts of time, and fixed waits either waste time or cut loading short.
• On bigger jobs, switch residential addresses between browser sessions, and check the address in use with our my IP tool.
• Always end with driver.quit() so Chrome closes and frees your computer's memory, and run headless on servers.
• Test the proxy with the proxy checker before a session, so a failure later on is never a mystery.

Common mistakes

• Reaching for Selenium first. If the API has the data, the browser route is slower and breaks more easily.
• Sharing code with your API key inside it. Anyone with the key can use up your quota. Keep it out of anything you publish.
• Forgetting to quit the browser. Each forgotten session leaves a copy of Chrome running in the background.
• Running out of quota on day one. Searches are the expensive part, so test with small numbers.

Beginner questions

Do I need a proxy for the API part?

No. The API answers based on your key and quota, so the example in Part 1 runs fine on your normal connection. Proxies matter for the browser part, where location changes what you see.

Why does a figure come back as None?

Not every figure is available for every video, and the code is written to cope with that. A missing figure comes back empty instead of stopping the script.

Can I download videos with this?

No, and this guide doesn't cover it. Both methods here read details about videos, such as titles and counts, within YouTube's rules.

Let the official API carry the structured data and keep Selenium for the few gaps, and a dependable YouTube project comes together without a fight. Keep your request rate steady, respect the Terms of Service, and let a good proxy handle location for the browser part. What you build from there is up to you.