Responsible Web Scraping: Rate Limits, robots.txt and Why Sites Block Bots

Every scraper runs on someone else's infrastructure. A careless crawler can slow a small business's website for real customers, rack up their hosting bill, or look indistinguishable from an attack. Responsible scraping isn't only ethics: it lowers legal risk, keeps your access stable and produces better data.

This guide covers the practical norms of polite scraping and explains why websites block bots, and what to do when it happens.

#Key takeaways

• Read and follow robots.txt, terms of service and any published crawl guidance.
• Rate-limit per domain, back off on errors and 429s, and adapt to server response times.
• Cache and scrape incrementally: never fetch what you already have.
• Identify your bot with a descriptive User-Agent and contact information.
• Bot protection is a signal, not a puzzle. If a site blocks you or presents a CAPTCHA, stop and seek permission or an official data source rather than trying to evade it.

#Understanding robots.txt

robots.txt sits at the root of a domain and tells crawlers which paths they may fetch:

textUser-agent: *
Disallow: /checkout/
Disallow: /account/
Crawl-delay: 5

User-agent: BadBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

• Disallow rules apply to matching user agents; more specific agent sections override *.
• Crawl-delay isn't part of the core standard and isn't honored by every crawler, but it's a clear statement of the site owner's wishes. Respect it.
• Sitemap points you to an efficient list of URLs. Use it instead of crawling blindly.

Check it in code before fetching (Python scraping tutorial shows how), or let your framework handle it (Scrapy's ROBOTSTXT_OBEY: Scrapy in production).

#Polite rate limiting

There's no universal "safe" rate, because it depends on the site's size. Principles:

• Start slow: for example, one request every one to few seconds per domain, and a low number of concurrent connections.
• Adapt: if response times increase, slow down. Scrapy's AutoThrottle does this automatically.
• Back off exponentially on errors (5xx, timeouts).
• Treat 429 Too Many Requests as a hard signal: honor Retry-After, and reduce your ongoing rate, not just for the next request.
• Spread load: crawl off-peak hours for the site's main audience, and interleave requests across domains rather than hammering one.

pythonimport time, random

class DomainThrottle:
    def __init__(self, min_delay=2.0):
        self.min_delay = min_delay
        self.last = {}

    def wait(self, domain: str):
        elapsed = time.monotonic() - self.last.get(domain, 0)
        delay = self.min_delay + random.uniform(0, 0.5)
        if elapsed < delay:
            time.sleep(delay - elapsed)
        self.last[domain] = time.monotonic()

    def penalize(self, factor=2.0, cap=60.0):
        self.min_delay = min(self.min_delay * factor, cap)   # after 429/5xx

#Cache and crawl incrementally

The most polite request is the one you don't make:

• Cache responses during development so you're not re-downloading while fixing parsers.
• Store content hashes and last-seen timestamps; revisit pages based on how often they actually change.
• Use sitemaps' lastmod and conditional requests (If-None-Match, If-Modified-Since) where supported. A 304 Not Modified costs the server very little.
• Download only what you need: skip images, fonts and media unless they're the data you're after.

#Identify yourself

Use an honest, descriptive User-Agent:

textAcmePriceMonitor/2.1 (+https://acme.example/bot-info; bot@acme.example)

A bot info page explaining what you collect, why, and how to contact you or opt out builds trust. Site operators who can reach you are far more likely to email than to block. Pretending to be a regular browser to avoid scrutiny undermines that trust.

#Avoid sensitive areas

• Don't scrape logged-in areas, carts, checkouts or account pages.
• Don't submit forms or trigger actions (search spam, fake sign-ups).
• Avoid personal data unless you've established a lawful basis (is web scraping legal?).

#Why websites block bots

Understanding the site owner's perspective explains why blocks happen:

• Infrastructure cost and stability: aggressive bots can make up a large share of traffic and slow the site for customers.
• Security: bots perform credential stuffing, card testing, inventory hoarding and vulnerability scanning. Defenses can't always tell a harmless scraper from an attacker.
• Business protection: companies may not want competitors or resellers extracting their pricing, content or data investment.
• Content licensing: publishers license content and protect it from unlicensed reuse, including for AI training.

Bot-management systems look at signals like request rates, IP reputation, browser characteristics and behavior patterns, and respond with rate limits, challenges, CAPTCHAs or blocks.

#What to do when you're blocked

A block, challenge or CAPTCHA means the site doesn't want that automated access, at least in that form. The responsible response is not to escalate with techniques to disguise or evade detection. Instead:

1. Stop the crawl for that domain.
2. Review your behavior: were you too fast, crawling disallowed paths, or hitting expensive endpoints like search?
3. Look for an official path: a public API, data feed, affiliate/partner program or data licensing (web scraping vs APIs).
4. Contact the site owner: explain your purpose, your rate limits and what you need. Many will agree to reasonable access, or offer an export.
5. Consider alternative sources: other providers, licensed datasets or data vendors with proper agreements.

Circumventing technical access controls can also create significant legal exposure, and it degrades trust across the whole ecosystem.

#A responsible scraping checklist

• Checked for an official API or data license first
• Reviewed terms of service and robots.txt
• Descriptive User-Agent with contact details and a bot info page
• Per-domain rate limits and concurrency caps
• Adaptive throttling and exponential backoff; 429s honored
• Caching, incremental crawling and conditional requests
• No logged-in areas, form submissions or personal data without a legal basis
• Stop-on-block policy, with escalation to seeking permission
• Monitoring of error rates to catch problems early

#FAQ

#Is ignoring robots.txt illegal?

robots.txt itself isn't a law, but ignoring it can be used as evidence of unauthorized or bad-faith access, and it breaks a widely accepted convention. Following it is standard practice for responsible crawlers.

#How many requests per second is acceptable?

It depends on the site. Large platforms handle more than small business sites. Start conservatively, respect any crawl-delay, adapt to response times, and stop if you see errors or 429s. If you need high volume, ask for permission or an API.

#Should I rotate User-Agents and IPs?

Using rotation to disguise a scraper and evade a site's blocks is not responsible practice and can increase legal risk. Legitimate large-scale crawling is typically done transparently, with permission, published bot information and cooperation with site owners.

#What if the data I need is only available by scraping?

Scrape it transparently and politely within the site's rules, collect only what you need, and reach out to the site owner. If access is blocked or prohibited, look for licensed alternatives rather than working around the restrictions.