How web crawlers find, fetch and respect the pages they visit

A web crawler is a program that visits web pages, records what it finds, follows the links on those pages and comes back later to see what changed. Search engines, web archives, SEO tools, price trackers and the pipelines that assemble AI training data all run crawlers. If you build agents or AI features, crawlers matter to you in three ways: the search results and datasets your agent relies on were probably collected by one, you may be tempted to write one, and an agent that fetches pages on its own looks a lot like a crawler to the people who run those sites.

The crawl loop in one screen

Strip away the scale and every crawler does four jobs: it keeps a queue of URLs to visit, fetches them, extracts content and new links from what it fetched, and decides when to come back. The queue is usually called the frontier. A minimal version looks like this:

frontier = load_seeds()
while frontier.has_due_urls() and budget.remaining():
    url = frontier.next_due()            # honours per-host spacing
    if not robots_allows(url, agent=UA):
        frontier.mark_skipped(url, reason="robots")
        continue
    resp = fetch(url, user_agent=UA, timeout=20)
    log(url, resp.status, resp.headers, fetched_at=now())
    if resp.is_html() and resp.is_indexable():
        store(url, extract_text(resp), content_hash(resp))
        for link in extract_links(resp):
            frontier.add(normalize(link))
    frontier.schedule_revisit(url, based_on=change_history(url))

Almost all of the real engineering hides inside those function names. next_due is where politeness lives. normalize decides whether two URLs are the same page. schedule_revisit decides how fresh your copy of the web will be. The rest of this page walks through each of them.

Discovery: where URLs come from

A crawler cannot fetch what it has never heard of. URLs enter the frontier from a handful of sources:

• Seed lists. A starting set chosen by hand: the sites you care about, a domain list, a directory.
• Links in fetched pages. The classic mechanism. Every fetched page adds its outgoing links to the queue, which is how a crawl spreads.
• Sitemaps. XML files, described at sitemaps.org, in which a site lists its own URLs, often with a last-modified hint. They are the cheapest way to find pages that nothing links to.
• Feeds. RSS and Atom feeds announce new items, which makes them useful for news, blogs and changelogs.

Discovery immediately creates a second problem: the same page often lives at many addresses. Tracking parameters, session identifiers, trailing slashes, upper and lower case, and http versus https all produce different strings for one document. A crawler normalizes URLs with explicit rules (strip known tracking parameters, lowercase the host, drop fragments) and respects rel="canonical" hints where a site declares its preferred address. Without this step a crawler spends much of its budget fetching duplicates.

Some sites also contain crawler traps: calendars that link to the next month forever, faceted search pages that combine filters into endless URL variations, or pages that generate fresh links on every request. Per-host URL caps, depth limits and pattern rules keep a crawl from disappearing into one of them.

Fetching without hurting the site

Politeness means the site owner should barely notice you. In practice that comes down to a few habits: a small number of concurrent connections per host, a pause between requests to the same host, crawling large sites in quieter hours, and asking for compressed responses. Conditional requests help both sides: if you send the ETag or last-modified time you saw before, an unchanged page can answer with a short 304 instead of the full document.

The response status tells a polite crawler what to do next:

Robots rules and other signals

Before fetching anything on a host, a well-behaved crawler reads /robots.txt. The format, known as the Robots Exclusion Protocol and published as RFC 9309, groups rules by user-agent and lists paths that are allowed or disallowed. The most specific matching rule wins.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: ExampleResearchBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Here every crawler may fetch the site except the cart and internal search results, while one named bot is asked to stay away entirely. Two misunderstandings are common. First, robots.txt is a request, not access control: nothing technically stops a crawler from ignoring it, which is exactly why respecting it is the baseline of good conduct. Second, it is not a security mechanism: a disallowed path is not secret, and sensitive content needs real authentication.

Other signals work at page level. A noindex robots meta tag or an X-Robots-Tag header asks search engines not to index a page, and nofollow asks them not to follow its links; these govern what happens after the fetch, not the fetch itself. Some site owners now publish separate rules for crawlers that gather AI training data, so check which user-agent tokens a site addresses rather than assuming one rule covers every purpose. Proposals such as llms.txt offer a curated summary of a site for language models; useful context, but not a permission system. And content behind a login or a paywall is outside a crawler's remit altogether.

Identity: say who you are

A crawler should be easy to identify and easy to contact. That means a stable user-agent string with a product token and a URL that explains what the bot does, which rules it follows and how to reach the people who run it:

User-Agent: ExampleResearchBot/1.2 (+https://example.com/bot)

Responsible operators go further. They publish the IP ranges their crawler uses or support reverse DNS checks, so a site owner can confirm that a request really came from them. They use different tokens for different purposes, such as building a search index, collecting training data, or fetching a single page because a user asked, so owners can allow one and refuse another. And they never disguise a crawler as an ordinary browser to slip past rules written for bots.

The same applies to agents. An agent that fetches a page in response to a user request is a different kind of visitor from a bulk crawler, and many operators label the two differently. Either way, the honest choice is to identify the traffic and respect what the site has asked for.

What happens to crawled data

Each fetch typically produces a record with the requested URL, the final URL after redirects, the status code, response headers, the time of the fetch, a hash of the content and the content itself. That content is either raw HTML or, for sites that build their pages with JavaScript, a rendered page; rendering needs a headless browser and costs far more than a plain fetch. Downstream steps remove boilerplate such as navigation and footers, detect language, collapse near-duplicates and split text into passages.

From there the data feeds very different uses: a search index, an archive, a training corpus, a change-monitoring service, a price comparison table or a research dataset. Two rules hold across all of them. The URL and fetch time should travel with the content, because a fact without its retrieval date cannot be judged for freshness. And crawling a page grants no rights to it: copyright, terms of use and personal data rules still apply to what you store and republish. The page on web scraping covers those boundaries in more detail.

Does your agent need a crawler?

Usually not. Most agent features need an answer to a question, not a copy of a website. A web search API gives your agent access to an index someone else maintains. Fetching a handful of known URLs on demand covers many monitoring and research tasks. A site's own API or data export is more stable than parsing its pages. Open crawl datasets such as Common Crawl exist for research at scale. Building your own crawler makes sense when you need defined coverage of specific sites at a known cadence, and you are ready to own the politeness, identity and storage duties that come with it.

Where GoCux fits

GoCux is not a crawler and does not read the open web. Its knowledge search works over the documents a project has imported. When web material is needed, it enters through a browsing or export step that someone authorized, and it is stored with a reference to the page it came from and the date it was retrieved, so a reviewer can judge both origin and age. Browser Link carries out a browser task only when that task has been explicitly granted; knowing a URL is not permission to get past a login. Exporting data is one of the actions that pauses for a decision by someone on the team. The note on web sources and responsible agent access and the crawler documentation describe this boundary.

Before you run a crawler or a fetching agent

• Is crawling really needed, or would a search API, a site API, an export or a feed answer the question?
• Does your user-agent name the bot and point to a page explaining it and how to contact you?
• Do you read and cache robots.txt per host and honour the group that matches your token?
• Are per-host concurrency, request spacing and back-off on 429 or 503 enforced in code?
• Do you normalize URLs and cap URLs per host to avoid duplicates and traps?
• Does every stored record keep its final URL, status and fetch time?
• Do you stop at 401 and 403 instead of looking for a way around them?
• Have you checked terms of use and personal data rules for what you plan to keep?