Scraping websites responsibly: methods, limits and better routes

Web scraping is the automated collection of specific information from web pages: a program, or more and more often an AI agent, opens a page, finds the values you care about and writes them into a table. Teams use it to track competitor prices, collect job postings, build prospect lists, watch regulatory notices and fill gaps in product catalogs. Agents have made scraping easier than ever, because you can now ask for data in plain language instead of writing code. That makes the old questions more pressing, not less: are you allowed to take this, is there a better route, and can you show where each value came from? This page is a practical overview, not legal advice.

Scraping, crawling and extraction

The three words are often used interchangeably, but they describe different jobs. Crawling is discovery at scale: following links to find and fetch many pages. Scraping is targeted: you know which pages matter and which fields you want from them. Extraction is the step that turns content into structured values, and it applies just as well to PDFs, filings and documents you already hold as to live web pages. A price-monitoring project, for example, might read a retailer's sitemap once to find product pages, scrape those pages weekly, and use extraction to normalize prices and sizes.

How a scraper works

1. Pick the targets. A defined list of pages or a URL pattern, with a reason for each.
2. Fetch the page. A plain HTTP request is enough for static pages. Pages that assemble their content with JavaScript need a headless browser that renders the page the way a person's browser would, which is slower and heavier.
3. Locate the values. Traditional scrapers use CSS selectors or XPath expressions that point at specific elements. AI-based scrapers read the page and pick out fields from a description such as "product name, list price, availability".
4. Clean and normalize. Convert currencies, units and date formats; strip whitespace and labels.
5. Validate. Reject values outside expected ranges, flag empty fields, compare with the previous run.
6. Store with provenance. Keep each value together with where and when it was read.
7. Watch for breakage. Websites change their layout without notice, and a broken selector usually fails silently by returning nothing or the wrong element.

Selector-based scrapers are precise but brittle. AI-based extraction copes better with layout changes, but it introduces a new failure: a confident value that is not on the page at all. The remedy is the same in both cases. Every value should arrive with the exact text it was taken from, and a value that could not be found should stay empty rather than be guessed.

{
  "product": "Standing desk, oak, 140 cm",
  "price": null,
  "price_status": "unknown: price shown only after login, not collected",
  "source_url": "https://shop.example.com/desks/oak-140",
  "retrieved_at": "time of the fetch, stored in UTC",
  "supporting_text": null,
  "method": "product page, price block selector v3"
}

Legal and ethical boundaries

Rules differ by country and by what you collect, so anything beyond a small internal experiment deserves a conversation with counsel. The same questions come up again and again:

• Terms of use. Many sites restrict automated access in their terms. If you log in, you have usually agreed to those terms, which can make them binding on you.
• Access controls. Logins, paywalls, CAPTCHAs and IP blocks are the site owner saying no. Getting around them is where scraping stops being a grey area and becomes clearly wrong, and in many jurisdictions potentially unlawful.
• Robots rules. A robots.txt file is rarely law, but it is a clear statement of what the owner wants automated visitors to do. Respect it.
• Personal data. Names, email addresses, photos and profile details remain personal data even when they are publicly visible. Data protection laws such as the EU's GDPR or Turkey's KVKK can apply to collecting, storing and using them, which raises questions of purpose, minimization and retention.
• Copyright and database rights. Facts such as a price are generally treated differently from expression such as an article, a review or a photo. Some jurisdictions also protect the investment that went into a database. Noting a price is not the same as republishing a page.
• Load. Aggressive scraping can slow down or knock over a small site. Throttle requests and identify your traffic honestly.

A useful ethical test: would you be comfortable if the site owner read your request logs, your user-agent string and your plan for the data?

Try these routes before scraping

Scraping is the right tool when none of these works and the use is clearly permitted. If you do go ahead, the guide to choosing a scraping API covers what to compare.

Keep provenance with every value

A scraped number without its source is a rumor. Provenance is what lets someone verify a value, settle a dispute, decide whether it is still current, respond to a removal request, or notice that a scraper broke three weeks ago. At minimum, store for each value the final URL after redirects, the time it was retrieved, the method or selector version that produced it, the exact text it came from, and a snapshot or hash of the page.

Write this into the acceptance criteria before anyone, person or agent, starts collecting:

Every price in the table links to the page it was read from and the time it was read. Each value quotes the text on the page that supports it. Pages that required a login, blocked the request or returned an error are listed with the reason and left empty. No value is inferred from a product name, a similar product or an earlier run.

Scraping with AI agents

When an agent does the collecting, the risks shift. An agent working in a logged-in browser session can see data its task never needed. It might decide on its own to work around a block. It might send collected data to another tool or another person. Treat these as governance questions: grant browser access per task, start read-only (see start agents read-only), require a person's decision before data leaves the workspace, and keep a trail of every page visited.

GoCux follows that model. It does not scrape or crawl the open web. Web material enters through a browsing or export step that someone authorized, and it is imported with a reference to the original page and the date it was retrieved. Browser Link carries out only the browser task that was explicitly granted, and a URL on its own is never permission to get past a login. Once the material is inside a project, Extraction pulls values from it, each paired with the passage that supports it, and a value that is not there stays unknown. Exporting data pauses for a decision, with the files in scope listed. See Extraction and web sources and responsible agent access.

Checklist for a scraping project

• Have you ruled out an API, an export, a feed or licensed data for this purpose?
• Have you read the site's terms of use and robots.txt, and noted what they allow?
• Does the plan avoid logins, paywalls and anything that blocks automated access?
• If personal data is involved, have you defined the purpose, the minimum fields and how long you will keep them?
• Are requests throttled, and does your traffic identify itself?
• Does every value carry its URL, retrieval time, method and supporting text?
• Will missing values be left empty and blocked pages reported, rather than filled in?
• Will someone review the data before it is shared outside the team?