Instagram data in Python: a responsible starter guide

In plain English: plenty of people would like Instagram numbers in a spreadsheet, such as how a public business profile describes itself, which hashtags are busy or how posts perform. Instagram keeps a tight grip on that data. Much of the app only works once you're signed in, its Terms of Use forbid unauthorised automated collection, and it pushes back against bots. The good news is that there's a proper way in. Meta, the company behind Instagram, offers the Instagram Graph API for exactly this. This guide explains that route first, sets out what fair collection of public data looks like, and then shows a small Python script, used with a proxy, that reads the basic public summary of a business profile and nothing more.

Quick start in 3 steps

1. If the account is yours, or you have its owner's permission, use the Graph API and skip scraping altogether.
2. Otherwise, set up the script below with your proxy details.
3. Fetch one public business profile, read its three summary labels and stop. No loops yet.

The rules before anything else

Instagram's Terms of Use rule out unauthorised automated data collection, and much of what makes the platform useful sits behind a login. Promise yourself these four things before you start:

• You'll only request content that's visible without signing in. A script must never run a logged-in account or try to slip past a login or challenge screen, because that breaks Instagram's terms.
• You'll respect robots.txt (the site's public list of areas that automated programs should stay out of), the Terms of Use and any rate limits.
• You'll leave personal information about individuals alone unless you have a lawful basis for it, and you'll follow privacy laws such as GDPR and CCPA.
• You'll keep the number of requests low and the pace gentle, so the service never feels your script at all.

If what you want can't be done inside those limits, scraping isn't the way to get it.

The Graph API: the route Instagram expects

For nearly every legitimate need, Meta's Instagram Graph API is where to go. It provides approved access to data from Business and Creator accounts, including media, comments, insights and hashtag data, already arranged in neat fields. Instead of reading web pages, your program presents an access token (a long, password-like key showing your app has permission) and calls documented addresses that return data. That means steady results and clear permission, with none of the fragile page-reading or the compliance risk. If you run the accounts you're studying, or their owners have agreed, the API is almost always the better tool. Give it serious thought before you go any further.

When reading public content is fair

Say you have a legitimate reason to note strictly public, non-personal facts, like the basic summary a business shows on its public profile. Then treat Instagram the way you'd treat any sensitive site: a minimal number of requests, generous gaps between them, and a proxy so no single address draws attention to itself. Instagram ties automated behaviour to IP addresses quickly, so residential proxies are the sensible fit here. Their addresses come from household broadband. Our residential premium option adds stability if you need it. All four proxy types are compared on the proxies overview, and the proxy for scraping page has advice for particular tasks.

What you need

• Python 3, plus somewhere to type commands.
• Proxy credentials: host, port, username, password.
• The usernames of a few public business profiles. Business profiles only, and only a few.

Step 1: Install and connect the proxy

pip install requests beautifulsoup4

Our proxies follow the common HOST:PORT format, with a username and password on top. Hand them to requests in a dictionary like this one, using your real details:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

Step 2: A fetcher that's careful by design

The function below turns a username into a profile address and asks for that one page. It sends ordinary browser details in its headers, waits for a single reply, then rests for a random 8 to 15 seconds. If the reply's status code isn't 200 (the code for "here's your page"), it stops on the spot. Reading the standard Open Graph labels from the page is far steadier, and far gentler, than trying to read a feed of posts.

import requests
import random
import time
from bs4 import BeautifulSoup

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch_public_profile(username):
    url = f"https://www.instagram.com/{username}/"
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=25)
    if resp.status_code != 200:
        print(f"Stopping: received status {resp.status_code}")
        return None
    time.sleep(random.uniform(8, 15))  # respectful pacing
    return resp.text

Step 3: Read the profile's public summary

Open Graph labels are bits of text that a page includes so that link previews in chat apps and on social media look right. On a public profile they hold a title, a description and an image link. Reading them touches nothing private and keeps your footprint tiny:

def parse_profile_meta(html):
    soup = BeautifulSoup(html, "html.parser")

    def meta(prop):
        tag = soup.find("meta", attrs={"property": prop})
        return tag["content"] if tag and tag.has_attr("content") else None

    return {
        "title": meta("og:title"),
        "description": meta("og:description"),
        "image": meta("og:image"),
    }

html = fetch_public_profile("example_business")
if html:
    print(parse_profile_meta(html))

Those three fields describe the business, not a person, and that's where to stay. If you're ever tempted to list someone's followers or gather details about individual people, that's your cue to stop and either switch to the Graph API or drop the idea.

Step 4: Back off, and save what you have

With Instagram, restraint is the whole method. A login page, a challenge page or any status other than 200 means you've reached a boundary, so stop straight away instead of retrying or switching addresses. Keep the total number of requests small, run the script only now and then, and store everything you collect so the same page is never requested twice. This helper writes each result as one line in a file called profiles.jsonl, together with the time it was collected:

import json
from datetime import datetime

def save_record(username, data, path="profiles.jsonl"):
    row = {"username": username,
           "fetched_at": datetime.utcnow().isoformat(),
           **data}
    with open(path, "a", encoding="utf-8") as f:
        f.write(json.dumps(row) + "\n")

Before you press run

Check that the script's traffic really goes through the proxy. Look up your own address on our my IP tool first, so you'd recognise it, and test the proxy endpoint with the proxy checker. Making sure your real address isn't leaking out is a basic step for every proxy project, not just this one.

Common mistakes

• Automating a login. It breaks Instagram's terms, whether you type the password into the script or borrow your browser's cookies.
• Missing a quiet login wall. Instagram may send you on to a sign-in page, and because requests follows redirects, the final status can still be 200. If the description comes back empty, or the title talks about logging in, treat it as a stop.
• Collecting people instead of businesses. Personal profiles and follower lists are personal data. Leave them alone.
• Looping before testing. Get one profile right before you even think about a second.

Beginner questions

Do I need a special account to use the Graph API?

The Graph API works with Business and Creator accounts. If you're studying your own account, the Instagram app lets you switch it to a professional account, which is the usual first step.

Why do I keep getting a login page?

Instagram shows a great deal of its content only to signed-in users. When a page asks you to log in, that's where the script stops. Getting a program to sign in for you breaks the terms.

If I use a proxy, is scraping Instagram fine?

No. A proxy changes which address Instagram sees. It doesn't change Instagram's rules, privacy law or what you're allowed to collect.

The responsible approach mirrors how Instagram itself is built. Use the official Graph API first, keep any reading of public pages small, polite and free of personal data, and stop at the first sign of friction. This code is a small, careful tool, not permission to collect at scale. Your judgement matters more than any line of it.