Scraping JavaScript-Heavy Websites With Playwright

Many modern websites are single-page applications: the initial HTML is nearly empty, and JavaScript fetches and renders the content. A plain HTTP request returns a shell with no data. For these sites, a headless browser (a real browser engine controlled by code) can render the page just like a user's browser.

Playwright is one of the best tools for this: fast, reliable, with auto-waiting and first-class Python and Node.js APIs. This guide shows how to use it for scraping efficiently and responsibly.

#Key takeaways

• Check for JSON APIs first: many JavaScript sites load data from endpoints you can observe in the Network tab.
• In Playwright, wait for specific elements or network responses, never fixed sleeps.
• Intercept API responses to capture clean JSON while the browser does the rendering.
• Block images, fonts and media to make headless scraping faster and lighter.
• Headless browsers are resource-heavy; limit concurrency and reuse browser instances.

#Do you actually need a browser?

Before reaching for Playwright:

1. Open DevTools → Network → Fetch/XHR and reload the page.
2. Look for requests returning JSON with the data you need.
3. Check the page source for embedded JSON (__NEXT_DATA__, window.__INITIAL_STATE__, JSON-LD).

If the data is available that way, and the site's terms allow it, a simple HTTP client is 10–100× cheaper (Python web scraping tutorial). Use a browser when content genuinely requires JavaScript execution or interaction.

#Setup

bashpip install playwright
playwright install chromium

#Basic scraping with Playwright

pythonfrom playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        user_agent="ResearchBot/1.0 (+https://example.com/bot; contact@example.com)",
        viewport={"width": 1366, "height": 900},
        locale="en-US",
    )
    page = context.new_page()
    page.goto("https://example.com/products", wait_until="domcontentloaded")

    page.wait_for_selector("[data-testid=product-card]")
    cards = page.locator("[data-testid=product-card]")

    products = []
    for i in range(cards.count()):
        card = cards.nth(i)
        products.append({
            "name": card.locator("h3").inner_text(),
            "price": card.locator("[data-testid=price]").inner_text(),
            "url": card.locator("a").first.get_attribute("href"),
        })

    browser.close()

#Waiting correctly

Flaky headless scrapers almost always come from poor waiting. Avoid time.sleep(5): it's both slow and unreliable.

#Intercepting API responses (the best trick)

Let the browser handle rendering, authentication tokens and request signing, while you capture the underlying JSON:

pythoncaptured = []

def on_response(response):
    if "/api/search" in response.url and response.status == 200:
        try:
            captured.append(response.json())
        except Exception:
            pass

page.on("response", on_response)
page.goto("https://example.com/search?q=laptops")
page.wait_for_response(lambda r: "/api/search" in r.url)

JSON is cleaner and more stable than scraping rendered HTML, and it often includes fields the UI doesn't show.

#Infinite scroll and "load more"

pythondef scroll_to_load(page, item_selector: str, max_rounds: int = 20):
    previous = 0
    for _ in range(max_rounds):
        count = page.locator(item_selector).count()
        if count == previous:
            break
        previous = count
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1200)   # brief pause for the next batch to request
    return previous

Better still, intercept the paginated API calls that infinite scroll triggers and page through them directly, if permitted.

#Block unnecessary resources

Images, fonts and video make up most page weight. Blocking them speeds up scraping and reduces load on the site:

pythonBLOCK = {"image", "media", "font"}

context.route("**/*", lambda route: route.abort()
              if route.request.resource_type in BLOCK else route.continue_())

Don't block scripts or XHR requests the page needs to render data.

#Concurrency without melting your machine (or the site)

Each browser page consumes significant memory and CPU. Use one browser, multiple contexts or pages, and a concurrency limit:

pythonimport asyncio
from playwright.async_api import async_playwright

async def scrape(urls, concurrency=4):
    sem = asyncio.Semaphore(concurrency)
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()

        async def worker(url):
            async with sem:
                page = await context.new_page()
                try:
                    await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
                    await page.wait_for_selector("h1", timeout=15_000)
                    return {"url": url, "title": await page.locator("h1").inner_text()}
                finally:
                    await page.close()
                    await asyncio.sleep(1.0)   # politeness delay

        results = await asyncio.gather(*(worker(u) for u in urls), return_exceptions=True)
        await browser.close()
        return results

Keep concurrency per target domain low, and respect rate limits (responsible web scraping).

#Reliability checklist

• Timeouts on navigation and waits, and catch TimeoutError per page.
• Retries with backoff for transient failures, and a cap on attempts.
• Screenshots and HTML snapshots on failure for debugging (page.screenshot(), page.content()).
• Close pages to avoid memory leaks; periodically restart the browser in long runs.
• Validate extracted data against a schema, and alert on empty fields (data quality guide).
• Run in containers with the right system dependencies, using the official Playwright images for consistency.

#Ethics and boundaries

A headless browser is a powerful tool; use it within the rules:

• Only scrape content you're permitted to access. Don't log in with accounts to scrape unless authorized.
• Don't use browser automation to bypass CAPTCHAs, bot-detection systems or access controls. When a site deploys these, it's signaling that automated access isn't welcome. Seek permission, a partnership or an official API instead.
• Keep request volumes modest and identify your bot where possible.

See is web scraping legal? for the legal landscape.

#Playwright vs Selenium vs Puppeteer

#FAQ

#Is Playwright good for web scraping?

Yes, for JavaScript-rendered sites. Its auto-waiting, network interception and resource blocking make headless scraping faster and more reliable. For static pages, a plain HTTP client is more efficient.

#Why is my headless scraper slow?

Browsers load a lot: scripts, images, fonts and third-party trackers. Block unnecessary resources, reuse browser instances, wait for specific conditions instead of full network idle, and capture API responses instead of parsing the DOM.

#Headless or headed mode?

Headless for production runs; headed (visible) mode for debugging selectors and flows. Playwright's trace viewer is also excellent for debugging failed runs.

#Can I run Playwright in the cloud or serverless?

Yes, in containers or serverless environments that support the browser's dependencies and memory needs. For large volumes, a worker pool of containers with a job queue is usually more predictable (scraping pipeline architecture).