Is Web Scraping Legal? A Practical Guide to the Ground Rules

"Is web scraping legal?" is one of the most searched questions about scraping, and the honest answer is: it depends. Scraping isn't inherently illegal, and it powers search engines, price comparison sites, academic research and archiving. But specific scraping activities can violate contracts, computer misuse laws, copyright, database rights or data protection law.

This article is general information, not legal advice. Laws differ by country and change over time. For commercial projects, especially involving personal data, consult a qualified lawyer.

#Key takeaways

• Scraping publicly accessible, non-personal data carries the least risk, but "public" doesn't mean "free to use however you like".
• Bypassing access controls (logins, paywalls, technical blocks) sharply increases legal risk.
• Terms of service can create contractual obligations, especially if you've agreed to them (for example, by creating an account).
• Copyright and database rights can restrict copying and reusing content, even when it's public.
• Personal data triggers privacy laws like the GDPR, regardless of whether the data is public.
• Low-risk practice: minimize data, respect robots.txt and rate limits, identify your bot, avoid personal data, and prefer official APIs or licenses.

#The main legal questions

A useful way to assess risk is to ask four questions:

1. How are you accessing the data? Public pages, or behind a login, paywall or technical barrier?
2. What data is it? Facts and prices, creative content, or personal data?
3. What did you agree to? Have you accepted terms of service that prohibit scraping?
4. What will you do with it? Internal analysis, republication, a competing product or AI training?

#1. Access: public vs protected

Computer misuse laws, such as the US Computer Fraud and Abuse Act (CFAA) and similar laws elsewhere, prohibit accessing computers "without authorization".

In the US, courts have narrowed how the CFAA applies to scraping of public data. The Supreme Court's Van Buren v. United States (2021) interpreted "exceeds authorized access" narrowly, and in the long-running hiQ Labs v. LinkedIn litigation, the Ninth Circuit indicated that scraping publicly accessible pages likely isn't "without authorization" under the CFAA. However, the same dispute ultimately turned on contract claims tied to LinkedIn's user agreement. Public-data scraping can still create liability through other legal theories.

Risk rises significantly when you:

• Scrape content behind a login using accounts, especially fake accounts or accounts created in violation of terms.
• Circumvent technical measures: CAPTCHAs, IP blocks after being blocked, paywalls, rate limits designed to stop you.
• Continue after receiving a cease-and-desist letter or explicit revocation of access.

#2. Terms of service and contracts

Many websites prohibit automated access in their terms. Whether those terms bind you depends on how they were presented and accepted:

• Clickwrap (you actively agreed, such as when signing up) is generally more enforceable.
• Browsewrap (terms linked in a footer that you never agreed to) is often weaker, depending on jurisdiction and circumstances.

If you have an account on a site and scrape while logged in, assume you're bound by its terms.

#3. Copyright

Facts themselves (a price, a temperature, a stock level) generally aren't protected by copyright. Expression is: articles, photos, reviews, product descriptions and creative compilations.

• Extracting facts for analysis is generally lower risk than copying and republishing content.
• Republishing copyrighted text or images can infringe, even if you credit the source.
• Exceptions like fair use (US) or text and data mining exceptions (in some jurisdictions, including the EU with conditions and opt-outs) may apply to certain uses. They're nuanced and fact-specific.
• Using scraped content to train AI models is an active and evolving area of law and litigation. Treat it with extra care.

#4. Database rights (EU and UK)

The EU and UK grant a sui generis database right to makers of databases involving substantial investment. Extracting or reusing a substantial part of such a database can infringe, even when individual items aren't copyrighted. This matters for scraping large catalogs, listings or directories in Europe.

#5. Personal data and privacy law

This is where many scraping projects get into trouble. Under the GDPR (EU/UK) and similar laws:

• Personal data is still personal data when it's public: names, emails, profiles, photos.
• You need a lawful basis (often "legitimate interests", which requires a documented balancing test).
• You must meet obligations for transparency, data minimization, purpose limitation, security, retention and data subject rights.
• Data protection authorities have fined companies for scraping and processing public personal data without adequate legal grounds or transparency.

Other regimes, such as California's privacy laws and sector rules, add their own requirements. If you don't need personal data, don't collect it.

#6. Other risks

• Trespass to chattels or similar claims if scraping burdens or harms a site's servers.
• Unfair competition claims in some jurisdictions when scraping is used to free-ride on a competitor's investment.
• Trademark and passing off if republished data misleads users about its source.

#A practical risk matrix

#Best practices that reduce legal risk

1. Look for an official API, data feed or license first (web scraping vs APIs).
2. Read and respect terms of service and robots.txt.
3. Stay on public pages. Don't log in to scrape unless you have permission.
4. Never circumvent technical access controls; if blocked, stop and ask (responsible web scraping).
5. Rate-limit so you never degrade a site's performance.
6. Identify your bot with a descriptive User-Agent and contact details.
7. Collect only what you need; avoid personal data, or run a privacy assessment if you must process it.
8. Transform, don't republish: use facts for analysis and insight rather than copying creative content wholesale.
9. Document your purpose, data inventory, legal basis and compliance decisions.
10. Respond promptly to complaints or opt-out requests.

#FAQ

#Is it legal to scrape publicly available data?

Often, but not always. Public availability reduces some risks (such as unauthorized-access claims in certain jurisdictions) but doesn't remove others, including contract, copyright, database rights and privacy obligations. Context and use matter.

#Does robots.txt have legal force?

robots.txt is a technical convention, not a law in itself. Ignoring it can still be used as evidence of bad faith or unauthorized access, and respecting it is widely considered best practice.

#Can I scrape data to train an AI model?

This is a fast-evolving legal area with ongoing litigation and new regulations, including opt-out mechanisms for text and data mining in some jurisdictions. Get specific legal advice, respect opt-outs, and prefer licensed or clearly permitted data.

#Can I scrape LinkedIn or other social networks?

Social networks typically prohibit scraping in their terms, host large amounts of personal data, and actively enforce against scrapers. The combined contract and privacy risks are significant. Use official APIs or partner programs where available.