Inside a search index: building it, refreshing it, knowing its limits

A crawler fetches pages; an index is what makes them findable. Every search engine, every web search API and most retrieval-augmented generation systems sit on top of one. If your agent searches anything, the index decides what it can find, how current that material is, and what it will never see. This page explains how an index is built, the two main structures inside modern indexes, and the trade-offs that shape every index you will rely on.

From fetched pages to searchable records

A raw crawl is a pile of HTML. Turning it into an index takes a pipeline, and each stage is a place where content can be lost or distorted:

1. Parse and clean. Extract the main text and drop navigation, cookie banners, footers and repeated boilerplate.
2. Deduplicate and canonicalize. Collapse identical and near-identical pages so one document is not indexed many times under different URLs.
3. Enrich. Detect the language, pull out the title, headings, publication hints and structured data, and record the fetch time.
4. Analyze the text. Split it into tokens, normalize case and apply language-specific rules such as stemming. For an agglutinative language like Turkish this step matters a great deal: sözleşme, sözleşmeler and sözleşmelerimizin should usually match each other.
5. Build index structures. Write the inverted index, the vector index, or both.
6. Attach ranking signals. Links between pages, anchor text, freshness, quality and spam signals, all used later to order results.

The output is a set of records, each one tied to a URL and a fetch time. Keep that last field in mind; it comes back when we get to freshness.

The inverted index

The classic structure maps each term to the list of documents that contain it. That list is called a posting list, and it usually stores positions and counts as well. A tiny example after analysis:

doc 1  "Renewal notice for supplier contracts"
doc 2  "Supplier onboarding checklist"
doc 3  "Contract renewal dates and notice periods"

term        posting list
renewal     1, 3
notice      1, 3
supplier    1, 2
contract    1, 3        (stemmed from "contracts")
onboarding  2
period      3           (stemmed from "periods")

A query for supplier renewal looks up two short lists instead of scanning every document. Document 1 contains both terms and ranks first; documents 3 and 2 each match one. Ranking functions such as BM25 refine this by rewarding terms that are rare across the collection and by stopping a long document from winning just because it repeats a word many times.

Inverted indexes are fast, cheap to update incrementally and easy to explain: you can always say which word matched. They excel at names, product codes, error messages, legal citations and rare terms. Their weakness is vocabulary. A query for cancel my plan will not find a page that only says terminate your subscription.

The vector index

A vector index stores an embedding for each document or passage: a list of numbers produced by a model so that texts with similar meaning sit close together. At query time the question is embedded the same way and the index returns its nearest neighbors. Comparing a query against every stored vector is too slow at scale, so vector indexes rely on approximate nearest-neighbor structures, graph-based ones such as HNSW or cluster-based ones, which give up a little accuracy for a large gain in speed.

Most serious systems now keep both and merge their results, an approach called hybrid search. How embeddings work and how the two rankings are combined is covered in semantic search.

Keeping an index fresh

An index is a snapshot, and the web keeps moving. Pages change at very different rates: a news front page changes many times a day, a product page changes when its price does, an old research paper may never change at all. Good crawlers learn each page's rhythm from its history and revisit accordingly, favoring pages that change often or matter more. Sitemaps with last-modified hints, feeds and push notifications from publishers all help spend visits where they count.

On the index side, most engines avoid full rebuilds. New and changed documents are written into new segments that are merged in the background, and deletions are recorded as markers until a later merge removes them. Removal is a freshness problem in its own right: a page that now returns 404, carries noindex or was taken down for legal reasons should leave the index promptly, or the index keeps serving answers that no longer exist anywhere.

Freshness also has to be visible. Each record should carry the time it was fetched, and that time should reach whoever consumes the result. An agent asked about current prices or a regulation now in force needs to see that a passage was retrieved a week ago, not merely that it matched.

Coverage, freshness and cost

Every index runs on a budget of fetches, storage and compute, and the budget forces choices:

• Breadth or recency. Visit more pages less often, or fewer pages more often.
• The long tail. Small sites, niche forums and documents in less common languages are expensive to cover relative to how often anyone asks about them.
• The hidden web. Content behind logins, forms, search boxes and paywalls is mostly absent, and pages that build their content with scripts may be indexed only partially.
• Depth of storage. Some indexes keep full text, others keep a snippet and a link, which limits what a downstream agent can quote.

The practical lesson for agent builders: "no results" usually means "not in this index", not "does not exist". An index that is temporarily unavailable is a different situation again, and a system that reports an outage as emptiness misleads everyone downstream. Likewise, a count of results is not proof that a collection is complete.

Building on someone else's index

When you call a web search API, you are renting an index. Questions worth asking any provider: how fresh is coverage in your domain, does each result carry its retrieval date, do you get full text or only snippets, how does the API signal truncated or partial results, and how are removals handled? Do not take the answers on trust; evaluate on your own queries. For some use cases you may not need a persistent vector store at all: search-based RAG retrieves fresh results at question time instead.

Project indexes in GoCux

GoCux does not maintain an index of the open web. Its knowledge search runs over the documents that belong to one project. Once a document is imported, full-text search works on it, and semantic retrieval adds to the results whenever it is available. Answers come back as passages with citations, so you can open the source. An empty result is treated as a legitimate answer, and "search unavailable" is reported as something different from "nothing found". Web material that a project imports keeps its original page reference and retrieval date, which keeps its age in plain view. And because memory is not a live feed, the way to get a fresh fact is to refresh the source, not to trust an old summary. See knowledge search and the guide to evaluating search.

Questions to ask about any index

• What exactly is in it: which sites, languages and document types, and what is deliberately left out?
• Does every result carry the time it was fetched, and do you pass that time on to the agent or user?
• Does it use keyword matching, vectors or both, and which one fails on your typical query?
• How are text analysis and stemming handled for each language you care about?
• How quickly are removed or changed pages reflected, and how would you find out?
• Can you tell an empty result from an unavailable index, and a truncated list from a complete one?
• Do you have a fixed set of test queries that you rerun whenever the index or its settings change?