Using LLMs for Web Scraping and Data Extraction

Traditional scrapers depend on CSS selectors and XPath: precise, fast and fragile. Change the layout and extraction breaks. Large language models offer a different approach: describe what you want ("the product name, price, currency and availability") and let the model find it, regardless of the page structure.

That's powerful for long-tail sources with thousands of different layouts. It's also slower, more expensive and probabilistic. This guide shows how to use LLMs for extraction effectively, and when not to.

#Key takeaways

• LLMs excel at extracting structured data from varied, messy or unfamiliar layouts.
• Use schema-constrained outputs and validate every record. Never trust raw model output.
• Clean and shrink the HTML before sending it to control cost and improve accuracy.
• The best production systems are hybrid: selectors or structured data for known high-volume sources, LLMs for the long tail and as a fallback.
• Measure accuracy on a labeled test set and track cost per record.

#Where LLM extraction shines

• Many sources, each with a different layout: local business sites, job postings across thousands of company career pages, event listings.
• Semi-structured text: specifications buried in paragraphs, policy terms, contact details in footers.
• Normalization and interpretation: "Ships in 3–5 business days" → {"min_days": 3, "max_days": 5}; categorizing products into your taxonomy.
• Resilience: small layout changes don't break extraction.

#Where it doesn't

• High-volume, single-source extraction where selectors work: an LLM call per page is far more expensive and slower than parsing.
• Exact numeric data at scale where tiny error rates are unacceptable, unless you add strong validation.
• Latency-sensitive paths.

#The extraction pipeline

textFetch page → Clean & reduce HTML → LLM extraction with schema → Validate → Store
                  │                                          │
                  └── structured data (JSON-LD) found? ──────┘ skip the LLM

#Step 1: Check for structured data first

JSON-LD, microdata or embedded JSON often contain exactly what you need, for free. Parse those before calling a model (Python web scraping tutorial).

#Step 2: Clean and reduce the HTML

Raw HTML is mostly noise: scripts, styles, navigation, SVGs, tracking attributes. Sending it all wastes tokens and distracts the model.

pythonfrom bs4 import BeautifulSoup
from markdownify import markdownify as md

def clean_for_llm(html: str, max_chars: int = 20_000) -> str:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup(["script", "style", "noscript", "svg", "iframe", "header", "footer", "nav"]):
        tag.decompose()
    main = soup.select_one("main, article, [role=main]") or soup.body or soup
    text = md(str(main), strip=["img"])            # Markdown keeps structure, drops noise
    return text[:max_chars]

Converting to Markdown or plain text with structure preserved often reduces token count dramatically while keeping headings, lists and tables readable.

#Step 3: Extract with a schema

pythonfrom pydantic import BaseModel, Field
from typing import Literal

class JobPosting(BaseModel):
    title: str
    company: str
    location: str | None = Field(description="City/region, or null if not stated")
    remote: Literal["remote", "hybrid", "onsite", "unknown"]
    salary_min: int | None = Field(description="Annual, in stated currency; null if absent")
    salary_max: int | None
    currency: str | None = Field(description="ISO 4217 code, e.g. USD")
    employment_type: Literal["full_time", "part_time", "contract", "internship", "unknown"]

PROMPT = """Extract the job posting from the page content below.
Use only information present in the content. If a field is not stated, use null or "unknown".
Do not guess salaries.

<page>
{content}
</page>"""

Use your LLM provider's structured output or tool-calling mode with this schema so responses always parse (prompt engineering for developers). Explicit "null if absent" instructions reduce fabricated values (reducing LLM hallucinations).

#Step 4: Validate

pythondef validate(job: JobPosting, page_text: str) -> list[str]:
    issues = []
    if job.salary_min and job.salary_max and job.salary_min > job.salary_max:
        issues.append("salary_range_inverted")
    if job.company.lower() not in page_text.lower():
        issues.append("company_not_found_in_page")      # cheap grounding check
    if job.currency and len(job.currency) != 3:
        issues.append("invalid_currency")
    return issues

Grounding checks (verifying that extracted strings actually appear in the source) catch many fabrications cheaply. Route records with issues to retry, a stronger model or human review.

#Hybrid: selectors + LLMs

A pragmatic production pattern:

1. Known high-volume sources → hand-written selectors or structured data. Fast and cheap.
2. Long-tail sources → LLM extraction.
3. Selector failure (required fields missing) → fall back to LLM extraction and alert that the selector needs updating.
4. LLM-generated selectors: ask the model to propose CSS selectors for a new source once, validate them on sample pages, then run them cheaply at scale. Re-generate when validation fails.

The fourth pattern gives you LLM flexibility at selector cost for recurring sources.

#Controlling cost

More in LLM cost optimization.

#Measuring accuracy

Build a labeled set of 100–300 pages across sources. Measure field-level accuracy, null accuracy (correctly leaving absent fields empty) and fabrication rate (values present in output but not on the page). Re-run on every prompt or model change (how to evaluate LLM applications).

#Agents that browse

Browser-using AI agents can navigate sites, click through pagination and fill search forms, which is useful for exploratory research and low-volume tasks (what are AI agents). For recurring, high-volume extraction, deterministic crawling plus LLM extraction is far cheaper and more predictable.

#Responsible use still applies

Using an LLM doesn't change the rules: respect terms of service, robots.txt and rate limits; avoid personal data without a lawful basis; and be careful with copyrighted content (is web scraping legal?). Also treat page content as untrusted input. Pages can contain text designed to manipulate LLMs (prompt injection), so don't give extraction models tools or privileges they don't need (AI guardrails and safety).

#FAQ

#Can LLMs replace traditional web scrapers?

For long-tail and varied sources, they can replace hand-written parsers. For high-volume single sources, selectors remain faster and cheaper. Most production systems combine both, often using LLMs to generate or repair selectors.

#Which LLM is best for data extraction?

Start with a smaller, cheaper model with structured-output support and measure accuracy on your labeled set. Move up to larger models only for fields or sources where the small model falls short.

#How do I stop the LLM from inventing data?

Instruct it to return null for absent fields, constrain outputs with schemas and enums, run grounding checks that extracted values appear in the source, and review low-confidence records.

#Is it expensive to use LLMs for scraping?

It can be if you send full raw HTML for every page. Cleaning content, caching by hash, using small models and generating reusable selectors typically reduces costs by an order of magnitude or more.