LinkedIn data with Python: the rules first, then the code

In plain English: LinkedIn is full of useful information: company pages, job adverts and short public profile summaries. People want it for job hunting, market research or finding business contacts. It's also one of the touchiest sites to collect data from. LinkedIn's User Agreement forbids unauthorised scraping (copying data with a program), much of what's there is about real people, which brings privacy law into play, and the site actively works against automated visits. So this guide runs in a deliberate order: the limits first, then LinkedIn's official tools, and only then a slow Python script for the narrow cases where reading a public page is reasonable.

Quick start in 3 steps

1. Check whether one of LinkedIn's official tools already gives you the data. Usually one does.
2. If you still need something from a public company page, set up the script below with your proxy details.
3. Fetch one page, read only its short public description, and stop at the first login screen or error.

Is this allowed? The limits come first

On LinkedIn, this section matters more than the code. Unauthorised scraping breaks its User Agreement, and collecting personal details at scale, or reaching anything behind a sign-in, can cause serious legal and privacy problems. Keep to these rules every time:

• Only open pages that anyone can see without signing in. Never let a script use a logged-in account, and never try to get around a sign-in screen. Both break LinkedIn's terms.
• Read LinkedIn's robots.txt file (where a site lists the areas automated programs should avoid) and its Terms of Service, and stay within them.
• Personal data needs a lawful basis before you collect or store any of it, and privacy laws such as GDPR and CCPA apply.
• Keep your request rate low, respect any limits, and never get in the way of the service for other people.

If any of these give you pause, take that as your answer and use an official, approved channel instead.

Try LinkedIn's official tools first

For most genuine needs, LinkedIn's own products are the right choice, and they're far safer. The LinkedIn Marketing and Talent Solutions APIs, LinkedIn's partner programme and Sales Navigator already cover much of what people try to scrape, with LinkedIn's blessing and in a consistent format. An API is a doorway that a company opens on purpose for programs, so the data arrives in a shape that doesn't keep changing, and, more importantly, clear permission to use it. Before you write a scraper, ask honestly whether one of these covers what you need. It usually does, and it saves you from both fragile code and legal risk.

When reading a public page is reasonable

Sometimes you have a fair reason to look at strictly public, non-personal information, such as a company's public "About" page or a job advert that's open to everyone. Then the method is the same as for any careful scraper: very few pages, long pauses and a proxy so the requests don't all come from one address. LinkedIn is quick to link automated patterns to the IP address behind them, so residential proxies (addresses from ordinary home internet connections) are the fitting type, and residential premium adds extra steadiness. You can compare every type on the proxies overview, and our proxy for scraping page has more advice on matching a proxy to a task.

What you need

• Python 3 with a command window you're comfortable in.
• A proxy login, copied from your dashboard.
• A short list of public company page addresses. Short really does mean short.

Step 1: Install, then add your proxy

pip install requests beautifulsoup4

Our proxies come in the standard HOST:PORT form with a username and password. requests takes them as a small dictionary, so swap the capitalised placeholders for your own details:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

Step 2: A fetcher that knows when to stop

This function asks for one public page through the proxy. Every reply comes with a status code, a number that says how the request went: 200 means "here's your page", while other numbers mean a redirect, a refusal or an error. The function stops politely on anything other than 200, and after a successful fetch it waits a random 8 to 15 seconds before the next one is allowed:

import requests
import random
import time
from bs4 import BeautifulSoup

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch_public_page(url):
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=25)
    if resp.status_code != 200:
        print(f"Stopping: received status {resp.status_code}")
        return None
    time.sleep(random.uniform(8, 15))  # generous, respectful pacing
    return resp.text

Notice how restrained it is: one page at a time, a long pause and an immediate stop when something looks wrong. That restraint is the point.

Step 3: Read the public description only

Public company pages carry Open Graph tags, small labels in the page's code that describe it for link previews on social media and in chat apps. Reading those three labels (name, description and address) is light and steady, and it avoids pulling anything personal:

def parse_company_meta(html):
    soup = BeautifulSoup(html, "html.parser")

    def meta(prop):
        tag = soup.find("meta", attrs={"property": prop})
        return tag["content"] if tag and tag.has_attr("content") else None

    return {
        "name": meta("og:title"),
        "description": meta("og:description"),
        "url": meta("og:url"),
    }

html = fetch_public_page("https://www.linkedin.com/company/example-co/")
if html:
    print(parse_company_meta(html))

Keep to descriptive, non-personal details like these. If you catch yourself writing selectors that collect individual people's names or profiles, stop and ask whether you should be collecting that at all.

Knowing when to stop

The most important habit on LinkedIn is backing off the moment you meet resistance. A redirect to a login page, a "check you're human" page or any status other than 200 is a clear signal to stop. Don't retry harder and don't switch addresses faster. Keep the total number of pages low, run the script rarely, and save what you collect so you never fetch the same page twice. A script that bows out early stays on the right side of LinkedIn and of the law.

Check your setup

Before running anything, make sure your traffic leaves through the proxy. Our my IP tool shows your own address for comparison, and the proxy checker tests the endpoint itself. Knowing your real address isn't slipping out is basic good practice for any proxy project.

Common mistakes

• Letting the script sign in. Copying your login cookies into a script counts too. It breaks LinkedIn's terms, so don't.
• Drifting into personal data. Company descriptions are one thing; lists of people are another, with legal duties attached.
• Retrying after a login wall. That page is the answer, not a hurdle.
• Thinking a proxy equals permission. It changes your address, not the rules.

Questions beginners ask

Is scraping LinkedIn legal?

There's no one-word answer. LinkedIn's User Agreement forbids unauthorised scraping, privacy laws cover any personal data, and the rules differ from country to country. We can't give legal advice. If you're unsure, use an official LinkedIn route or don't collect the data.

Will a proxy stop LinkedIn noticing my script?

That's the wrong job for a proxy. It changes the IP address LinkedIn sees, but it doesn't make anything allowed that wasn't allowed before, and it isn't a way around LinkedIn's rules. Use it only to keep a very small number of polite requests from all coming out of one address.

I only want my own LinkedIn data. Do I need any of this?

No. LinkedIn's settings include an option to download a copy of your own account data, which is quicker and fully within the rules.

For LinkedIn the honest summary is short: start with the official tools, keep any public-page reading tiny and polite, and stay well away from logged-in pages and personal information. The rules are strict for good reasons, and the approved route nearly always lasts longest.