Python Web Scraping Tutorial: requests, httpx and BeautifulSoup

Learn web scraping with Python step by step — fetching pages with requests and httpx, parsing with BeautifulSoup, pagination, JSON endpoints, async scraping, retries and saving data.

By Danyal Ahmed - Data Scientist · August 12, 2026 · 3 min read

Python is the most popular language for web scraping, thanks to a mature ecosystem and readable code. This tutorial walks through building a real scraper, from a single page to a polite, concurrent crawler with retries and validated output.

We'll use httpx for HTTP (a modern alternative to requests, with async support) and BeautifulSoup for parsing. Before scraping any site, read its terms and robots.txt, and keep request rates low (is web scraping legal?).

Key takeaways

Use an HTTP client with timeouts, retries with backoff and an honest User-Agent.
Parse with CSS selectors targeting stable attributes; prefer embedded JSON or JSON-LD where available.
Handle pagination explicitly and deduplicate URLs.
Use async with a concurrency limit and per-domain delays for speed without overloading sites.
Validate records and store them in structured formats (CSV, JSON Lines, Parquet or a database).

Setup

bashpython -m venv .venv && source .venv/bin/activate
pip install httpx beautifulsoup4 lxml pydantic tenacity

Step 1: Fetch a page

pythonimport httpx

HEADERS = {
    "User-Agent": "BookResearchBot/1.0 (+https://example.com/bot; contact@example.com)",
    "Accept-Language": "en-US,en;q=0.9",
}

client = httpx.Client(headers=HEADERS, timeout=httpx.Timeout(20.0), follow_redirects=True)

resp = client.get("https://books.toscrape.com/")
resp.raise_for_status()
print(resp.status_code, len(resp.text))

We use books.toscrape.com, a sandbox site built for practicing scraping.

Always set a timeout. Without one, a hung connection can block your scraper indefinitely.
Reuse a client (connection pooling) instead of calling httpx.get repeatedly.

Step 2: Parse HTML with BeautifulSoup

pythonfrom bs4 import BeautifulSoup

soup = BeautifulSoup(resp.text, "lxml")

for card in soup.select("article.product_pod"):
    title = card.select_one("h3 a")["title"]
    price = card.select_one(".price_color").get_text(strip=True)
    rating = card.select_one("p.star-rating")["class"][1]   # e.g. "Three"
    link = card.select_one("h3 a")["href"]
    print(title, price, rating, link)

Tips:

select and select_one use CSS selectors, which are readable and powerful.
get_text(strip=True) removes surrounding whitespace.
Access attributes like a dict: tag["href"], or tag.get("href") to avoid KeyErrors.

Step 3: Resolve URLs and paginate

pythonfrom urllib.parse import urljoin

def parse_listing(html: str, base_url: str):
    soup = BeautifulSoup(html, "lxml")
    items = [urljoin(base_url, a["href"]) for a in soup.select("article.product_pod h3 a")]
    next_link = soup.select_one("li.next a")
    next_url = urljoin(base_url, next_link["href"]) if next_link else None
    return items, next_url

url = "https://books.toscrape.com/catalogue/page-1.html"
product_urls = []
while url:
    r = client.get(url)
    r.raise_for_status()
    items, url = parse_listing(r.text, str(r.url))
    product_urls.extend(items)

product_urls = list(dict.fromkeys(product_urls))   # dedupe, preserve order

urljoin converts relative links (../page-2.html) into absolute URLs.

Step 4: Extract detail pages into a schema

Define what a valid record looks like. Validation catches silent extraction failures.

pythonfrom decimal import Decimal
from pydantic import BaseModel, HttpUrl

class Book(BaseModel):
    url: HttpUrl
    title: str
    price: Decimal
    currency: str
    in_stock: bool
    upc: str

def parse_book(html: str, url: str) -> Book:
    soup = BeautifulSoup(html, "lxml")
    table = {row.th.get_text(strip=True): row.td.get_text(strip=True)
             for row in soup.select("table.table-striped tr")}
    price_text = soup.select_one(".product_main .price_color").get_text(strip=True)
    return Book(
        url=url,
        title=soup.select_one(".product_main h1").get_text(strip=True),
        price=Decimal(price_text.lstrip("£")),
        currency="GBP",
        in_stock="In stock" in table.get("Availability", ""),
        upc=table["UPC"],
    )

If the markup changes and a field disappears, Pydantic raises an error instead of letting bad data through.

Step 5: Look for JSON before parsing HTML

Many modern sites embed data as JSON, which is far more stable than HTML:

pythonimport json

def extract_json_ld(soup: BeautifulSoup) -> list[dict]:
    blocks = []
    for tag in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(tag.string or "")
            blocks.extend(data if isinstance(data, list) else [data])
        except json.JSONDecodeError:
            continue
    return blocks

products = [b for b in extract_json_ld(soup) if b.get("@type") == "Product"]

Also check the browser's Network tab for XHR/fetch calls returning JSON. If the page loads data from an endpoint, requesting that endpoint directly is simpler and lighter than rendering the page, as long as it's permitted by the site's terms. For sites where the data truly requires rendering, see scraping JavaScript websites with Playwright.

Step 6: Retries with exponential backoff

pythonfrom tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception

def is_retryable(exc: Exception) -> bool:
    if isinstance(exc, httpx.HTTPStatusError):
        return exc.response.status_code in (429, 500, 502, 503, 504)
    return isinstance(exc, (httpx.TimeoutException, httpx.TransportError))

@retry(stop=stop_after_attempt(4), wait=wait_exponential_jitter(initial=2, max=30),
       retry=retry_if_exception(is_retryable))
def fetch(url: str) -> httpx.Response:
    r = client.get(url)
    r.raise_for_status()
    return r

Respect 429 Too Many Requests and any Retry-After header. If you're getting 429s, slow down permanently, not just for one retry.

Step 7: Polite async scraping

For many pages, async I/O speeds things up. Limit concurrency and add delays so you don't overload the site:

pythonimport asyncio, random

SEM = asyncio.Semaphore(3)          # max 3 concurrent requests
DELAY = (1.0, 2.5)                  # seconds between requests per worker

async def fetch_book(aclient: httpx.AsyncClient, url: str) -> Book | None:
    async with SEM:
        await asyncio.sleep(random.uniform(*DELAY))
        r = await aclient.get(url)
        if r.status_code != 200:
            return None
        return parse_book(r.text, url)

async def main(urls: list[str]):
    async with httpx.AsyncClient(headers=HEADERS, timeout=20, follow_redirects=True) as aclient:
        results = await asyncio.gather(*(fetch_book(aclient, u) for u in urls), return_exceptions=True)
    return [b for b in results if isinstance(b, Book)]

books = asyncio.run(main(product_urls))

Step 8: Save the data

pythonimport json

with open("books.jsonl", "w", encoding="utf-8") as f:
    for b in books:
        f.write(b.model_dump_json() + "\n")

JSON Lines is append-friendly and robust for large outputs. For analysis, load it into pandas and write Parquet; for ongoing pipelines, upsert into a database keyed by a stable ID (here, upc).

Step 9: Check robots.txt programmatically

pythonfrom urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://books.toscrape.com/robots.txt")
rp.read()
allowed = rp.can_fetch(HEADERS["User-Agent"], "https://books.toscrape.com/catalogue/page-2.html")

Where to go next

For hundreds of thousands of pages, use a framework with scheduling, deduplication and pipelines built in: Scrapy in production.
For messy, varied layouts, consider LLM-powered extraction.
For long-running systems, design for monitoring and change detection: scraping pipeline architecture.

FAQ

requests or httpx?

Both are excellent for synchronous scraping. httpx adds native async support and HTTP/2, which helps with concurrent scraping. If you already use requests and don't need async, it's fine.

BeautifulSoup or lxml?

BeautifulSoup offers a friendly API and can use lxml as its fast parser backend. For very large-scale parsing, lxml or selectolax directly are faster. Choose readability first, then optimize if parsing is a bottleneck.

How fast can I scrape a site?

As slowly as is reasonable for the site's size and your needs. Start with roughly one request per second or less per domain, watch for errors or 429 responses, and back off. Check whether the site publishes a crawl-delay or has an API.

Why does my scraper get different HTML than my browser?

The content may be rendered by JavaScript, personalized by cookies or location, or served differently based on headers. Compare the raw response with what you see in the browser, check the Network tab for API calls, and use a headless browser only if necessary.

Python Web Scraping Tutorial: requests, httpx and BeautifulSoup

#Key takeaways

#Setup

#Step 1: Fetch a page

#Step 2: Parse HTML with BeautifulSoup

#Step 3: Resolve URLs and paginate

#Step 4: Extract detail pages into a schema

#Step 5: Look for JSON before parsing HTML

#Step 6: Retries with exponential backoff

#Step 7: Polite async scraping

#Step 8: Save the data

#Step 9: Check robots.txt programmatically

#Where to go next

#FAQ

#requests or httpx?

#BeautifulSoup or lxml?

#How fast can I scrape a site?

#Why does my scraper get different HTML than my browser?