Why Your Scraper Got Blocked — and the Polite Pattern That Doesn't

Blocked scrapers usually earned it

"My scraper got blocked" is told as a war story — the site fought back, the arms race began. Read the details and it is usually a confession. The script never fetched robots.txt. It requested forty pages in ten seconds from one IP. Its User-Agent claimed to be Chrome on Windows while it was Python in a loop. Some were reaching behind logins for data the site never made public.

Sites block that traffic because it is indistinguishable from abuse — because, mechanically, it is abuse, whatever the intent. The frustrating part is that most legitimate monitoring needs none of it. If what you want is "check a handful of public pages once a day and tell me what changed", you can do that politely, and polite scrapers watching a few pages daily rarely get blocked, because they look like what they are.

Rule 1 — fetch robots.txt before anything else

robots.txt is the site telling you, in a machine-readable file at a standard address, exactly what it is willing to have crawled and how fast. Most "blocked scraper" stories start with skipping it.

Honouring it is barely code: fetch /robots.txt, check your target paths are allowed, and if a Crawl-delay is declared, wait that long between requests. If the file disallows your path, the answer is no — not "no unless I rotate proxies". A monitor built to refuse disallowed pages never starts the arms race, which is the only reliable way to win it.

Rule 2 — rate-limit like a human, not like a loop

A person checking a competitor's pricing page opens it, reads it, maybe opens another. A naive script requests everything it knows about at machine speed — which is precisely the signature rate-limiters and WAFs exist to catch.

The polite version waits seconds between requests, checks each page once a day rather than once a minute, and spreads its schedule instead of firing at midnight sharp. For competitor price monitoring this costs you nothing: prices change daily at most, so checking faster than daily buys no information — it only buys risk.

Rule 3 — identify yourself honestly

Faking a browser User-Agent feels clever and works until anyone looks. A request that claims to be Chrome but never loads images, executes no JavaScript, and arrives on a metronome is more suspicious than a request that says plainly what it is.

An honest User-Agent — a name and a way to reach you — is both the ethical choice and the durable one. Site operators routinely tolerate polite, honest bots they can identify and rate-limit; it's the liars they ban on sight.

Rule 4 — public pages only, ever

Everything above applies to pages anyone can open in a private browser window. Behind a login the situation changes completely: you agreed to terms of service, the data is not public, and both the legal and ethical footing disappear. No competitive question is worth automating a logged-in session to answer. A well-built monitor doesn't make this a policy you remember — it makes it a thing the code cannot do.

What a polite monitor actually looks like

Wire those habits to a small skeleton — fetch politely → extract a value → append it to history → alert on change — and you have a monitor that runs unattended for months. The extraction layer deserves one extra courtesy to your future self: a primary CSS selector per page, with a currency-pattern fallback so a site redesign degrades your data instead of silently killing it.

Build it once, understand it forever

You could clone someone's finished scraper — and be back at rule 1 the first time it breaks, maintaining code you never understood. We took the opposite route and packaged it as Build a Price Monitor with Claude Code: a build-along guide containing every prompt, in order, that produced a working, live-tested monitor — watchlist, robots-aware fetching, price history, webhook alerts — plus the finished repo as your answer key. Chapter 0 is the ethics above, enforced in code; the pattern rebuilds in an evening for job postings, stock levels, or any public page that changes.

Either way, steal the rules. Fetch robots.txt first, slow down, tell the truth about who you are, and stay on the public side of every login. The sites you monitor won't notice you — which is exactly the point.