Web Scraping and the Law: What You Can and Cannot Do

This is not legal advice. The article gives a general orientation and does not account for the facts of any specific case. Laws change, and how they apply depends on the country, the purpose and how the data is used. If money or reputation depends on the answer, talk to a lawyer.

"Is scraping legal?" sounds like a simple question, but there is no single answer. Collecting information from an open page is, in itself, what you do when you read it and type it into a spreadsheet. The questions come from what you collect, how, and what you do with it afterwards. Here are five areas to look at. We cover the EU and Russia because those are the rules our users most often ask about; other countries have their own.

1. The site's terms of use

Many sites explicitly ban automated collection or commercial reuse in their terms. How binding that is for you depends on whether you accepted the terms (for example, when signing up) and on local law. The practical takeaway is the same everywhere:

• read the terms before collecting from a site regularly;
• if you are signed in, you almost certainly accepted an agreement, and it applies to you;
• if the terms forbid collection and you really need the data, ask the owner — many offer an export, a partner programme or an API.

2. robots.txt is a signal, not a lock

The robots.txt file at the root of a site tells automated programs which sections the owner asks them not to crawl. The protocol is standardised in RFC 9309, which says plainly that these rules are not a form of access authorization: they are rules crawlers are requested to honour.

How to treat it:

• robots.txt is written mainly for search crawlers that visit sites in bulk on their own;
• an extension you run by hand in an open tab is a different case, but the file still shows how the owner feels about automated access;
• if a section is disallowed in robots.txt and the terms ban collection, that is a strong reason to stop.

3. Personal data

This is the most important part. If your spreadsheet contains names, phone numbers, emails, photos or profiles of people, you are most likely processing personal data — and it does not matter that the data was published openly.

European Union: GDPR

If the data relates to people in the EU, or you operate in the EU market, the GDPR applies:

• personal data is any information about an identified or identifiable person; being public does not take data outside the regulation;
• you need one of the lawful bases in Article 6 — for scraping the usual candidate is "legitimate interests", which requires weighing your interests against people's rights;
• Article 14 requires telling people that you obtained their data from someone other than them: who you are, why you process it, where it came from and what rights they have. The deadline is a reasonable period and at most one month, or at the first contact if you use the data to reach them. Exceptions are narrow;
• maximum fines are up to €20 million or 4% of worldwide annual turnover.

Russia: Federal Law 152-FZ

Russia's personal data law defines personal data as any information relating directly or indirectly to an identified or identifiable individual. Whoever organises the processing is an "operator" with legal duties.

The key provision for scraping is Article 10.1, in force since 1 March 2021. Before that, the law had a category of "publicly available" personal data, often read as permission to reuse anything posted online. Courts disagreed, and the law changed:

• there is now a category of data the person has permitted for dissemination, which requires a separate consent in which the person can set bans and conditions;
• if a person published data themselves without such consent, whoever further distributes or processes it must prove that the processing is lawful;
• the person can demand that transfer of their data stop.

Operators also have general duties: a purpose, a legal basis, a processing policy, in many cases notifying the regulator (Roskomnadzor), and storing databases of Russian citizens' data on servers in Russia. Penalties have been tightened in recent years.

In practice

• Collect only the fields you need. If market analysis needs company name and city, leave out personal names and phone numbers.
• Separate companies from people. A company is not a data subject, but a director's name or a manager's mobile number is personal data.
• Do not use scraped contacts for bulk outreach without a legal basis; marketing and e-privacy rules apply on top of data protection law.
• Store data carefully and delete it when the purpose is met.

4. Copyright and database rights

Individual facts — a price, a product name, an address — are generally not protected by copyright. Descriptions, photos and reviews often are. Collecting them for internal analysis and republishing them on your site are very different things.

Databases have their own protection:

• In the EU, Directive 96/9/EC creates a sui generis right that protects against extracting or re-using all or a substantial part of a database whose making involved substantial investment.
• In Russia, Article 1334 of the Civil Code gives the database maker an exclusive right to extract materials when making the database took substantial investment. Unless shown otherwise, a database of at least 10,000 independent elements is presumed to qualify. Extraction means transferring all or a substantial part of the contents to another medium.

In practice: pulling a couple of hundred items to compare prices is not the same as copying a large service's whole catalog and launching a competing directory on top of it.

5. Load on the site

Even when data is open and contains nothing personal, frequent requests put load on someone else's server. Good practice:

• pauses between pages comparable to a human's pace;
• limited volume — collect what you need, not "everything just in case";
• no circumventing protections: captchas, blocks and closed sections signal that the owner does not want automated access. Getting around technical protection measures or into closed areas is a different legal territory altogether.

How Grabsavo handles this

We designed the extension so that responsible collection is the default:

• it runs in your browser with your session and sees only what you see;
• a random 2–4 second pause between pages by default, an optional page cap, and an automatic stop when no new rows appear;
• item pages are visited no more than two at a time;
• no proxies, no captcha solving, no bypassing sign-ins or paywalls;
• page data is not sent to Grabsavo servers. The exception is the optional AI assistant, which receives a cleaned sample of 2–3 list items without form values.

Responsibility for what you collect and how you use it stays with you, as with any tool.

Short checklist

1. I read the site's terms and robots.txt.
2. I collect only the fields my purpose needs.
3. If there is data about people, I know my legal basis and how I will inform them or delete it.
4. I do not copy someone's whole database or republish their texts and photos.
5. I kept the delays and limited the volume.

Try it

Practise delay and limit settings on our demo catalog — it is our own practice page, so collect as much as you like. Install Grabsavo from /install.

Sources

• Regulation (EU) 2016/679 (GDPR), Article 14 — gdpr-info.eu
• Directive 96/9/EC on the legal protection of databases — EUR-Lex
• RFC 9309, Robots Exclusion Protocol — RFC Editor
• Federal Law No. 152-FZ "On Personal Data" (in Russian) — ConsultantPlus
• Overview of the Article 10.1 amendments (in Russian) — Anti-Malware.ru
• Civil Code of the Russian Federation, Article 1334 (in Russian) — zakonrf.info