Fresh web data in RAG: acquisition, freshness and citations

Retrieval-augmented generation over your own documents is a fairly well understood problem: the corpus changes slowly and you control it. Point the same pipeline at the web and two things change. The corpus moves underneath you, and many questions gain a time dimension. "What does the vendor's pricing page say" really means "what does it say now", and a confident answer built from last quarter's copy is wrong in a way that is hard to notice. This guide walks through the parts that differ when the data is live: deciding how fresh each answer must be, acquiring pages, recording provenance, chunking for citation, ranking with time in mind, and refreshing without silent drift.

Decide what "live" has to mean

Freshness is not one requirement. Group the questions you expect by how quickly their answers change and how much a stale answer costs:

That table points to three architectures. Query-time retrieval calls a search API, fetches the top results and hands extracted text to the model. It is fresh by construction, but slower, harder to reproduce and dependent on someone else's ranking. Ingest-time retrieval acquires pages ahead of time, indexes them and answers from the index. It is fast and repeatable, but only as fresh as the last refresh. Most production systems end up hybrid: a maintained index for the sources you know matter, plus query-time search for the long tail. The trade-off between the first two is covered in search-based RAG without a vector database; the rest of this guide applies to both.

Acquisition: take the boring, reliable route

Use the most structured channel a site offers before reaching for a scraper:

• Feeds and sitemaps. RSS or Atom feeds, and XML sitemaps with lastmod values, tell you what is new without fetching everything.
• Official APIs and data downloads. Cleaner data, often with explicit terms of use.
• Search APIs. For discovery when you do not know the URLs in advance. See what a web search API is.
• Fetching known URLs. With a headless browser only where the content needs JavaScript to appear.

Then extract the main content, not the whole page. Navigation, cookie notices and "related stories" rails poison retrieval because they repeat across pages and match many queries. The article extraction guide covers boilerplate removal and metadata. Respect robots rules and site terms throughout. A source you are not allowed to store is not a candidate for a persistent index, even if a query-time fetch would technically work.

Provenance travels with every chunk

In a live-web index, the most important question about a retrieved passage is not whether it is relevant but where and when it came from. Store that on every chunk, not only on the document:

{
  "chunk_id":      "page-key:pricing/team-plan:2",
  "url":           "https://vendor.example.com/pricing?ref=nav",
  "canonical_url": "https://vendor.example.com/pricing",
  "retrieved_at":  "YYYY-MM-DDThh:mm:ssZ",
  "published":     null,
  "modified":      "YYYY-MM-DD",
  "heading_path":  ["Pricing", "Team plan"],
  "content_hash":  "sha256:...",
  "source_type":   "vendor page",
  "status":        "current",
  "text":          "..."
}

retrieved_at is the honest timestamp: the page said this at that moment. modified is the site's own claim, which may be missing or wrong. status separates current chunks from superseded ones and from pages that have since disappeared. Without these fields you cannot answer "when was this true", and neither can the model.

Chunk for citation, not only for recall

Most chunking advice optimizes retrieval quality. With web data, optimize for a second goal as well: every chunk should be quotable and easy to locate.

• Split on structure. Use headings, list boundaries and table edges before falling back to fixed sizes. A pricing table cut in half produces confident wrong answers.
• Carry context into the chunk. Prefix each chunk with the page title and heading path, so "the Team plan includes" still makes sense on its own.
• Keep identifiers stable. Derive chunk IDs from the canonical URL, heading path and position, so a refresh that changes one section does not renumber the whole page.
• Drop repeated template text shared across a site before indexing, or it will crowd out real content in results.
• Keep offsets. Record where the chunk sits in the extracted text, so a cited sentence can be shown in its original context.

Retrieval and answers that know what time it is

Similarity search has no sense of time. Add it explicitly:

• Filter out superseded and removed chunks by default, and keep them queryable for questions about history.
• For time-sensitive question types, prefer recent retrieval dates when relevance is close, and set a maximum age beyond which a chunk is not used without a fresh fetch.
• Put the retrieval date next to each passage in the prompt, and instruct the model to state dates when it answers time-sensitive questions.
• When sources disagree, show both with their dates instead of letting the model choose silently.

For citations, require every claim in the answer to point to a chunk, and check mechanically that any quoted text actually appears in that chunk. A link that resolves is not the same as a passage that supports the claim. When nothing relevant is retrieved, the right answer is "no source found", not a fluent guess. Grounding to reduce hallucinations covers that failure mode in more depth.

Refresh without silent drift

A live index starts to decay the moment it is built. Treat refresh as a first-class job:

1. Set the cadence per source, from the freshness table, rather than one schedule for everything.
2. Use cheap change checks first. Conditional HTTP requests with ETag or Last-Modified, sitemap dates and feed entries tell you what changed without full downloads. Compare content hashes after extraction, because many pages change only their ads.
3. Re-chunk and re-embed only what changed, and mark old chunks superseded instead of deleting them, so past answers stay explainable.
4. Handle disappearance. A page that now returns 404 or redirects elsewhere should be marked removed, not left serving stale text.
5. Invalidate cached answers that cited chunks which have changed.
6. Watch the refresh itself. Alert when a source fails to refresh, when extraction suddenly returns far less text, or when a site's structure changes. From the answer side, a refresh job that has been failing for weeks looks exactly like a healthy one.

Some changes deserve a person's attention rather than an automatic re-index, such as an edited regulation or a revised contract term. For those, a reviewed monitoring step works better than a silent update; see setting a baseline before you monitor.

Where GoCux fits

GoCux is not a web search API and does not read the open web, so it is not the acquisition layer of a live-web RAG pipeline. Its Knowledge search answers a question within one project's own documents and returns passages with their citations. Full-text search works once material has been imported, and semantic retrieval adds to it when available. It also keeps apart two outcomes that pipelines often blur: an empty result is a legitimate answer, while an unavailable search is a different state and is reported as such.

Web pages reach a project only through an authorized browsing or export step, and on import they keep their original page reference and retrieval date, which gives you the provenance fields above. For questions that must stay current, Pulse runs a recurring question over sources you choose and puts each change in front of you for review instead of quietly overwriting what you had. Memory is not a substitute, because it is not a live feed; fresh facts come from refreshed sources. For the rules on which sources work may draw on, see the source policy documentation.

Checklist before you go live

• Every question type has a freshness requirement and an architecture: query time, pre-indexed or hybrid.
• Each source has an acquisition channel, permission to store, and a refresh cadence.
• Main content is extracted, and site templates are removed before indexing.
• Every chunk carries URL, canonical URL, retrieval time, site dates, heading path, hash and status.
• Superseded and removed chunks are filtered by default but kept for history.
• Answers state dates for time-sensitive claims, and quotes are checked against chunks.
• "No source found" is an allowed answer.
• Refresh failures and extraction anomalies raise alerts with a named owner.