Most scraping projects reach the legal team late, as a request to sign off on something already built. That is the slowest way to get approval. The questions lawyers ask are predictable, and a team that answers them up front usually gets a decision in days instead of weeks.
This checklist sets out the questions, grouped the way a legal review usually runs, with why each one matters and what a good answer looks like. It is written for the engineers and data leads who bring the project, so they know what to prepare.
This is general information, not legal advice. Rules differ by country and change quickly, and your own counsel decides what applies to your project.
1. What data, exactly?
The first questions are about scope, and vague answers here slow everything else down.
- Which fields will you collect? List them. “Product data” is not an answer; “product name, price, currency, stock status and URL” is.
- From which sites, and how many pages? Name the sources, the volume and the frequency.
- What is it for? The purpose decides much of what follows: internal price monitoring, a product feature, resale, or training an AI model each raise different issues.
- Do you need all of it? Expect to be asked to drop fields that do not serve the purpose.
A good answer is a one-page scope with a sample record and a field list, each field tied to the purpose.
2. Is any of it personal data?
This is often the most important question, and the one teams get wrong most often by assuming that public means unrestricted.
- Could any field identify a person, alone or combined with others? Names, usernames, photos, profile URLs and contact details all count. So do reviews and posts with an author attached.
- Whose data is it, and where do they live? European and UK data protection law applies to personal data of people in those regions, wherever the collector is based, and it makes no exception for data that is publicly available.
- What is the legal basis, and has anyone weighed the project’s interest against the people’s rights?
- How will people be told? When data is collected about people rather than from them, the law still expects them to be informed, and regulators have fined companies that built databases from publicly posted material.
- Can you avoid it? The best answer is often to exclude personal fields at collection time.
A good answer shows which fields are personal, why each is needed, and how the rest are filtered out before storage. If personal data is central to the project, expect a formal impact assessment. Our GDPR compliance overview covers the basics.
3. Where does it come from?
Legal will want to know exactly what kind of access the project relies on.
- Is everything publicly visible without logging in? Data behind a login is a different matter: collecting it usually means accepting the site’s terms, and US cases have turned on exactly that. In the hiQ and LinkedIn dispute, hiQ won early rulings on the anti-hacking law but was later found to have breached the user agreement, partly through fake accounts, and the case ended with an injunction and an order to destroy the data.
- Have you accepted the site’s terms? If an account or a clicked agreement is involved, legal will read those terms. A US court held in 2024 that Meta’s terms did not bar logged-off collection of public data, which shows why the logged-in and logged-out cases are treated so differently.
- What do robots.txt and other opt-out signals say? Even where robots.txt is not legally binding, legal will ask whether you respect it. For AI training in the EU, machine-readable reservations carry legal weight. Our guide to robots.txt and AI opt-outs covers the signals.
- Is there an official API or a licence? If a site offers its data through an API or for a fee, legal will ask why you are not using that route.
A good answer lists each source with its access type (public, logged in, API), its terms and its robots.txt position, checked on a stated date.
4. How will you collect it?
The method matters, separately from the data. Claims based on how a site was accessed have survived in court even where claims about the data itself failed: in X Corp’s 2024 case against a data company, the court dismissed the claims about scraping and reselling public posts but later let claims about server burden go forward.
- How fast, and how much load? State the request rate per site and how it adapts to errors and slow responses.
- Does anything get around an access control? Logins, paywalls and technical measures that restrict access are where legal risk rises sharply. Expect a firm question here.
- How do you identify yourselves? Legal may ask whether the collector names itself or provides a contact.
- Who provides the infrastructure? If you use a proxy network or a scraping service, legal will ask how the provider sources its IP addresses and what it logs. Our explainer on how providers ethically source residential IPs covers what to ask a vendor.
A good answer describes the collection setup in plain terms: rates, retry and backoff behaviour, how blocks are handled, and a statement that the project does not bypass logins or paywalls.
5. Who owns it, and what can you do with it?
Collecting data does not make you its owner. Legal will separate what you can freely use from what stays someone else’s.
- Is it facts or content? Prices, dates and counts are generally not protected by copyright. Article text, photos and the wording of reviews are.
- Does it come from a European database? In the EU and UK, the maker of a database who invested substantially in it can stop others extracting a substantial part, even when each item is a plain fact.
- Will you share, sell or publish it? Internal analysis, a customer-facing feature and resale each carry more risk than the last.
- Will it train an AI model? Expect questions about opt-outs, the source of every part of the dataset, and how acquisition was documented. Recent US rulings have treated how training material was acquired as a separate question from how it was used.
A good answer separates the dataset into facts, derived data and copied content, and explains what leaves the company. Our guide to who owns scraped data covers these layers in depth.
6. Where will it live, and for how long?
- Where is it stored and processed? Jurisdiction affects which laws apply, and the location of your proxy exits is not the same as the location of your data. See data residency is not proxy location.
- Who can access it? Expect questions about access control and security, especially for anything personal.
- How long will you keep it? Have a retention period and a way to delete data when it ends.
- Can you remove a specific record or source on request? If a person or a site owner asks, you need to be able to find and delete what came from them.
A good answer names the storage location, the access group, the retention period and the deletion process.
7. Who is accountable, and what if someone objects?
- Who owns the project? A named person who can answer questions and stop collection.
- What is documented? Sources, dates, settings and changes, kept so that you can show later what was collected and how. Monitoring helps here too; see how to monitor a web scraping pipeline.
- What happens if a site sends a complaint or a cease-and-desist letter? Legal will want a plan: who receives it, how quickly collection from that source can be paused, and who decides what happens next.
- When will the project be reviewed again? Sites change their terms and signals; a yearly review, or one whenever the purpose changes, is common.
A good answer names the owner, points to the documentation, and describes a stop procedure that can be executed the same day.
A one-page intake summary
Bring this to the first meeting, filled in. It answers most of the questions above before they are asked.
| Topic | What to bring |
|---|---|
| Scope | Field list, sample record, sources, volume, frequency |
| Purpose | What the data is for, and who will use it |
| Personal data | Which fields are personal, why they are needed, how others are excluded |
| Access | For each source: public or logged in, terms, robots.txt, API available or not |
| Method | Request rates, backoff, block handling, no bypassing of logins or paywalls |
| Infrastructure | Proxy or scraping vendor, and how it sources IPs |
| Use | Internal, customer-facing, resale or AI training |
| Storage | Location, access group, retention period, deletion process |
| Accountability | Project owner, documentation, complaint and stop procedure, review date |
A standing policy makes this faster for every project after the first. Our data collection policy template turns these answers into company-wide rules, so each new project only has to document what is different. For the wider legal picture, see is web scraping legal.
FAQ
Does scraping need legal approval?
Many companies require it, especially for projects that involve personal data, logged-in access, resale or AI training. Even where it is not required, a short review avoids building something that later has to be taken down.
What is the most common reason legal blocks a scraping project?
Personal data without a clear purpose and legal basis is a frequent one. Collection behind logins, and plans to republish copyrighted content, are others. Narrowing the scope often resolves all three.
Is public data free to scrape?
Not without conditions. Public personal data is still protected by data protection law in Europe and the UK, creative content is still protected by copyright, and in the EU some databases are protected even when they contain only facts. How you collect it also matters.
How long does a legal review of a scraping project take?
It depends on the company and the project, but most of the delay comes from back-and-forth on scope and data. Bringing a completed intake summary to the first meeting removes most of it.
The bottom line
Legal teams ask the same questions about every scraping project: what data, whose data, from where, how, for what, stored where, and who is accountable. Prepare the answers before the review starts, keep personal data and logged-in access out of scope unless they are essential, and document sources and methods as you go. Projects that arrive with a clear intake summary are approved faster, and they are easier to defend if anyone asks questions later.