Architecture of a Large-Scale Web Scraping Pipeline

A scraper that handles 500 pages on a laptop and a system that reliably processes millions of pages a day from thousands of sites are different animals. At scale, the hard problems are scheduling, politeness, failure handling, data quality and knowing when something breaks. This guide lays out an architecture that handles all of them.

#Key takeaways

• Separate the system into stages: frontier/scheduler → fetchers → raw storage → parsers → validation → normalized storage.
• Store raw responses so you can re-parse without re-fetching.
• Enforce politeness per domain centrally, not per worker.
• Make every stage idempotent and connected by queues for resilience and scaling.
• Monitor data, not just infrastructure: fill rates, record counts and extraction errors per source.

#The reference architecture

text                ┌────────────────────┐
Seeds/sitemaps ─▶ URL frontier &      │  priorities, dedupe, recrawl schedule,
                │ scheduler           │  per-domain politeness
                └─────────┬──────────┘
                          ▼  fetch jobs (queue)
                ┌────────────────────┐
                │ Fetcher workers     │  HTTP clients; headless browser pool
                │ (stateless)         │  for JS-only pages
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │ Raw store           │  object storage: HTML/JSON + metadata
                └─────────┬──────────┘
                          ▼  parse jobs (queue)
                ┌────────────────────┐
                │ Parser workers      │  per-source extractors (selectors, JSON, LLM)
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │ Validation &        │  schema checks, normalization, dedupe,
                │ normalization       │  entity matching
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │ Serving storage     │  database / warehouse / search index
                └────────────────────┘
       Cross-cutting: monitoring · alerting · config registry · access logs

Each box scales independently. Queues between stages absorb bursts and let you retry a stage without redoing the others (message queues explained).

#1. The URL frontier and scheduler

The frontier decides what to fetch next:

• Discovery: seeds, sitemaps, category pages and pagination.
• Deduplication: normalized URLs (lowercase host, removed tracking params, sorted query strings) plus a seen-set, via a database index or a Bloom filter at very large scale.
• Prioritization: important pages first (high-value products, frequently changing pages).
• Recrawl scheduling: revisit pages based on observed change frequency. A page that hasn't changed in 30 fetches can wait longer; volatile prices get checked more often.
• Politeness: per-domain rate limits and concurrency caps, robots.txt rules and crawl-delays. Enforce these centrally, so adding workers never increases load on any single site (responsible web scraping).

A common implementation uses per-domain queues with a "next allowed fetch time", which the scheduler checks before dispatching jobs.

#2. Fetchers

Fetchers are stateless workers that take a job, perform the request and write the result:

• HTTP clients with timeouts, retries with backoff and connection reuse.
• A separate headless browser pool only for sources that need rendering. Route by source configuration, because browsers are expensive (scraping with Playwright).
• Honest User-Agent, conditional requests (ETag/Last-Modified) and response size limits.
• Emit metrics: status codes, latency and bytes per domain.

#3. Raw storage

Write every successful response to object storage with metadata:

texts3://raw/{source}/{yyyy}/{mm}/{dd}/{url_hash}.html.gz
metadata: url, fetched_at, status, headers, content_hash, fetcher_version

Why this matters:

• Re-parse without re-fetching when you fix a parser bug. This saves time and avoids re-hitting sites.
• Change detection via content hashes: skip parsing when nothing changed.
• Debugging and auditing: see exactly what the page looked like.

Apply retention limits appropriate to your legal and privacy obligations.

#4. Parsers

Parsing is decoupled from fetching and organized per source (or per template):

• Structured first: JSON-LD, embedded state JSON, API responses.
• Selectors for HTML, maintained with fixtures and tests (Scrapy in production shows testing patterns).
• LLM extraction for long-tail sources with varied layouts (using LLMs for web scraping).

Version your parsers. Record the parser version with every extracted record so you can trace and reprocess.

#5. Validation and normalization

Treat every record as untrusted:

• Schema validation: required fields, types, ranges (a price must be a positive number; a date must parse).
• Normalization: currencies, units, time zones, text encoding, whitespace.
• Deduplication and entity resolution: the same product across sources, or the same listing reposted. Use identifiers (GTIN, SKU, URL canonicalization) and fuzzy matching where needed.
• Quarantine invalid records with reasons, rather than silently dropping or storing them.

See data quality guide for the broader discipline.

#6. Serving storage

Choose by access pattern:

• PostgreSQL for current-state records with upserts.
• Append-only history tables or a warehouse for time series (price history, availability over time).
• Search indexes for text search across records.
• Analytics in a warehouse or lakehouse (data warehouse vs lake vs lakehouse).

#7. Monitoring: the part that makes it production-grade

Infrastructure metrics tell you workers are up. They don't tell you the data is right. Monitor per source:

Alert owners when a source deviates from its baseline, and give them tools: the raw page, the parser version and a failing-record sample.

#8. Operating model

• Source registry: configuration per source (URLs, rate limits, fetch mode, parser, owner, legal notes).
• Runbooks for common failures: layout change, block, site outage.
• Fast parser deployment: fixing a broken source should take minutes to hours, not days.
• Compliance records: terms reviewed, personal-data assessment and opt-out handling (is web scraping legal?).

#Scaling considerations

• Horizontal scaling of fetchers and parsers via queue consumers (horizontal vs vertical scaling).
• Backpressure: when parsers or databases fall behind, the scheduler should slow dispatch.
• Cost control: prefer HTTP over browsers, skip unchanged pages, compress raw storage and tier old data to cheaper storage.
• Idempotency everywhere: re-running a fetch or parse job produces the same result, and upserts are keyed by stable IDs.

#FAQ

#What queue or orchestration tool should I use?

For job queues, managed queues or Redis/RabbitMQ-based job systems work well; Kafka suits very high-volume event streams. For scheduled batch crawls tied to data pipelines, an orchestrator like Airflow can trigger and track runs (building data pipelines with Airflow).

#Should I store raw HTML?

In most cases, yes, for a limited retention period. It enables re-parsing, debugging and change detection. Weigh it against storage costs and any legal or privacy constraints on retaining the content.

#How do I detect that a site changed its layout?

Monitor per-source record counts, field fill rates and validation failures against historical baselines. Sharp deviations almost always mean a layout change. Parser tests with fresh fixtures confirm it.

#How many pages per day can one worker handle?

It depends on politeness limits, response times and whether you need a browser. HTTP fetchers can handle many requests across many domains concurrently; browser workers are far heavier. In practice, per-domain politeness, not worker capacity, usually determines throughput.