Scrapy in Production: Building Scalable, Maintainable Crawlers

Once a scraping project grows beyond a few hundred pages, hand-rolled scripts start reinventing the same things: request queues, retries, rate limiting, deduplication, output handling. Scrapy is a mature Python framework that provides all of that, built on an asynchronous engine that crawls efficiently while staying polite.

#Key takeaways

• Scrapy structures scraping into spiders (what to crawl and how to parse), items (data shape) and pipelines (validation, cleaning, storage).
• AutoThrottle, concurrency limits and robots.txt compliance are built in. Turn them on.
• Use item validation and stats monitoring to catch silent breakage.
• Test parsers against saved HTML fixtures so layout changes are caught quickly.
• Scrapy is ideal for HTML and API crawling at scale; combine it with a headless browser only for pages that truly need JavaScript.

#Project setup

bashpip install scrapy
scrapy startproject catalog
cd catalog
scrapy genspider books books.toscrape.com

textcatalog/
  catalog/
    spiders/books.py     # spider logic
    items.py             # item definitions
    pipelines.py         # validation & storage
    middlewares.py       # request/response hooks
    settings.py          # politeness, concurrency, pipelines
  scrapy.cfg

#A spider with pagination and detail pages

pythonimport scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    def parse(self, response):
        for href in response.css("article.product_pod h3 a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_book)

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_book(self, response):
        rows = {r.css("th::text").get(): r.css("td::text").get()
                for r in response.css("table.table-striped tr")}
        yield {
            "url": response.url,
            "title": response.css(".product_main h1::text").get(),
            "price": response.css(".product_main .price_color::text").get(),
            "availability": rows.get("Availability"),
            "upc": rows.get("UPC"),
        }

response.follow resolves relative URLs, and Scrapy's scheduler deduplicates requests by default, so the same URL isn't fetched twice in a run.

Run it and export:

bashscrapy crawl books -O books.jsonl

#Politeness settings (turn these on)

python# settings.py
BOT_NAME = "catalog"
USER_AGENT = "CatalogResearchBot/1.0 (+https://example.com/bot; bot@example.com)"
ROBOTSTXT_OBEY = True

CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 1.0                 # base delay between requests to the same site

AUTOTHROTTLE_ENABLED = True          # adapts delay to server response times
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]

HTTPCACHE_ENABLED = True             # great during development
DOWNLOAD_TIMEOUT = 30

AutoThrottle slows down automatically when the server responds slowly, which is exactly the behavior a considerate crawler should have. See responsible web scraping.

#Items and validation pipelines

Define the shape of your data and validate it in a pipeline:

python# pipelines.py
from decimal import Decimal
from scrapy.exceptions import DropItem

class CleanAndValidatePipeline:
    def process_item(self, item, spider):
        if not item.get("title") or not item.get("upc"):
            spider.crawler.stats.inc_value("items/dropped_missing_fields")
            raise DropItem(f"Missing required fields: {item.get('url')}")

        item["price"] = Decimal(item["price"].replace("£", "").strip())
        item["in_stock"] = "In stock" in (item.get("availability") or "")
        return item

class DedupePipeline:
    def __init__(self):
        self.seen = set()

    def process_item(self, item, spider):
        if item["upc"] in self.seen:
            raise DropItem("duplicate")
        self.seen.add(item["upc"])
        return item

pythonITEM_PIPELINES = {
    "catalog.pipelines.CleanAndValidatePipeline": 100,
    "catalog.pipelines.DedupePipeline": 200,
}

For stricter validation, use Pydantic models or dedicated item-validation libraries. The goal is to drop and count bad items, not silently store them.

#Storing results

• Feeds (FEEDS setting): export to JSON Lines, CSV or Parquet, locally or to object storage.
• Database pipelines: upsert items into PostgreSQL keyed by a stable identifier.
• Message queues: push items to a queue for downstream processing (scraping pipeline architecture).

pythonFEEDS = {
    "s3://my-bucket/books/%(time)s.jsonl": {"format": "jsonlines", "encoding": "utf8"},
}

#Middlewares

Downloader middlewares let you modify requests and responses globally: adding headers, handling specific status codes, routing certain URLs to a headless browser renderer, or recording metrics. Scrapy integrates with Playwright via plugins for pages that require JavaScript. Use them selectively, only for URLs that need it (scraping with Playwright).

#Incremental crawling

For recurring crawls, don't refetch everything every time:

• Use sitemaps with lastmod dates (SitemapSpider).
• Store previously seen URLs and content hashes; revisit on a schedule matched to how often pages change.
• Honor HTTP caching headers (ETag, Last-Modified) where servers support them.

This reduces load on the target site and your costs.

#Testing spiders

Save real HTML pages as fixtures, and unit-test your parse methods:

pythonfrom scrapy.http import HtmlResponse, Request

def fake_response(path, url):
    body = open(path, "rb").read()
    return HtmlResponse(url=url, body=body, encoding="utf-8", request=Request(url=url))

def test_parse_book():
    spider = BooksSpider()
    item = next(spider.parse_book(fake_response("tests/fixtures/book.html",
                                                "https://books.toscrape.com/x")))
    assert item["upc"] == "a897fe39b1053632"
    assert item["title"]

When the site changes, save a new fixture, watch the test fail, and fix the parser. Contracts (docstring-based checks) are another lightweight option built into Scrapy.

#Monitoring

Scrapy collects stats per run: requests, responses by status code, items scraped, items dropped, errors. Export them to your monitoring system and alert on:

• Item count dropping sharply versus previous runs.
• Rising 4xx/5xx or 429 rates (slow down!).
• Drop rates in validation pipelines, often the first sign of a layout change.
• Run duration anomalies.

#Deployment

Options range from simple to managed:

• A cron job or scheduler running scrapy crawl in a container.
• An orchestrator such as Airflow triggering crawls as part of data pipelines (building data pipelines with Airflow).
• Scrapyd or hosted Scrapy platforms for managing many spiders.
• Distributed crawling with shared queues when a single machine isn't enough.

#FAQ

#Scrapy vs BeautifulSoup: which should I use?

BeautifulSoup is a parser; Scrapy is a complete crawling framework (scheduling, concurrency, retries, pipelines, export). For a few pages, requests plus BeautifulSoup is simpler. For large or recurring crawls, Scrapy saves a lot of work.

#Can Scrapy scrape JavaScript websites?

Not natively. It fetches raw responses. You can often use the site's underlying JSON APIs directly, or integrate a headless browser via plugins for specific pages.

#How do I make Scrapy faster?

Increase concurrency carefully, disable unneeded middlewares, use efficient selectors, and avoid rendering pages unnecessarily. But speed must never come at the expense of politeness: per-domain limits and AutoThrottle should remain in place.

#Does Scrapy respect robots.txt?

Yes, when ROBOTSTXT_OBEY = True, which new projects enable by default. Keep it on.