Search API or your own vector index: choosing retrieval for RAG

The default first design for retrieval-augmented generation is a vector database: chunk your documents, embed them, store the vectors, and run a similarity search at question time. It works, and it also hands you a pipeline to maintain. For a large class of questions there is a simpler option: send the question, or a rewritten form of it, to a search API, fetch the top results and give the model the extracted text. This page is about choosing between the two, and about the hybrid designs most teams end up with. It assumes you know the basics of semantic search and of web search APIs.

Two pipelines, side by side

An owned index pipeline acquires documents, extracts text, chunks, embeds and stores. At question time it embeds the query, retrieves the nearest chunks, optionally reranks them, and generates. You own the corpus, the ranking and the failure modes.

A search-API pipeline does its work at question time. It rewrites the question into one or more queries, calls the search API, fetches and extracts the top pages (or uses the snippets or page text the API returns), optionally reranks, and generates. Someone else owns the index and the ranking. You own query formulation, page selection and what you do with the text.

"Owned index" does not have to mean vectors only. A keyword index with BM25 ranking, or a hybrid of keyword and vector scores, is often a better first owned index than a pure vector store, especially for names, codes and exact phrases. The real choice is between an index you build and one you rent.

When a search API is the better retriever

• The corpus is the open web. You cannot index the web yourself in any meaningful sense, and a provider that already crawls it covers far more ground. See what a web index is.
• Freshness matters. News, product pages and announcements change faster than most teams can re-crawl and re-embed. The guide to RAG with live web data covers freshness in detail.
• Questions are broad and unpredictable. If you cannot list the sources in advance, you cannot pre-index them.
• You are early. A search-backed prototype can tell you whether the feature is useful at all before you invest in ingestion, chunking strategy and index operations.
• You would rather not hold copies of third-party content. Fetching at question time and discarding afterwards avoids building a long-lived copy of other people's pages, though terms of use still apply to what you fetch.

When your own index is the better retriever

• The content is private. Internal documents, contracts, tickets and wikis are invisible to public search. This alone settles most company use cases.
• Access control matters. An owned index can filter by who is asking. Document-level permissions are hard to enforce through someone else's ranking.
• You need repeatability. The same question should retrieve the same passages tomorrow, so an answer can be audited or a regression test can pass. Public search results shift as the index and ranking change.
• The corpus is stable and well defined. A product's documentation, a set of regulations or a research collection changes in known versions you can re-index deliberately.
• The vocabulary is specialized. Part numbers, legal citations and internal project names often rank poorly in general search and well in a tuned keyword index.
• Volume and cost must be predictable. Per-query charges grow with traffic; an owned index has mostly fixed costs plus engineering time. Which is cheaper depends entirely on your volumes, so model both with your own numbers.
• Questions must not leave your systems. Sending every question to an external provider may be unacceptable for sensitive work.

A comparison to fill in for your case

Fill the table in for your own situation, then score candidates on real queries rather than demos. Evaluating web search on your own queries describes how to build that test set.

Hybrid designs that hold up

Systems that last usually combine the two. Four patterns cover most needs:

1. Route by question. Simple rules or a small classifier send questions about internal matters to the owned index and questions about the outside world to search. Log every routing decision so misroutes can be reviewed.
2. Retrieve from both, then rerank. Query both, merge with rank fusion or a reranking model, and label each passage with its origin so the answer can say which came from where.
3. Search, then cache. Pages fetched through search become a short-lived index with a time to live. Repeated questions hit the cache; expired entries are fetched again. This also leaves a record of exactly what the model saw.
4. Curate, then fall back. Keep an owned index of the sources you trust most, such as official documentation or regulators, and use search only when that index has nothing relevant, restricted to an allowlist of domains where possible.

def retrieve(question, user):
    route = classify(question)          # "internal", "external" or "both"
    passages = []
    if route in ("internal", "both"):
        passages += owned_index.search(question, allowed_for=user)
    if route in ("external", "both"):
        for page in search_api.query(rewrite(question), domains=ALLOWLIST):
            passages += extract_and_chunk(page, retrieved_at=now())
    ranked = rerank(question, passages)
    log_retrieval(question, route, ranked)   # what the model actually saw
    return ranked[:TOP_K]

The logging line is not optional. Without a record of what was retrieved, a search-backed answer cannot be explained after the fact, because by then the search results will have moved on.

What you give up, and how to get it back

Search-backed retrieval trades away control. Three habits recover most of it.

First, decide which sources may answer: domain allowlists or blocklists per question type, written down rather than implied. The note on deciding which sources may answer makes the case for being explicit.

Second, snapshot what you use: store the extracted text, URL and retrieval time of every page that contributed to an answer, so the answer can be checked later against what the model was actually given.

Third, treat empty as an answer: when neither source returns anything relevant, say so instead of letting the model fill the gap from memory.

Where GoCux fits

GoCux sits on the owned-corpus side of this choice. It is not a web search API and does not read the open web. Its Knowledge search takes a question scoped to a single project's documents and returns passages with their citations. Full-text retrieval works as soon as material is imported, and semantic retrieval contributes when it is available. An empty result counts as a valid answer, and a search that cannot run is reported as unavailable rather than as "nothing found", a distinction worth copying in any pipeline.

When web material is needed, it comes in through an authorized browsing or export step and keeps its page reference and retrieval date, so it joins the project's corpus with provenance intact. Research runs from an approved plan with a source allowlist, and an allowlist left empty permits no source at all. For how to judge retrieval quality, see the documentation on evaluating search.

Decision checklist

• List your question types and mark which need private content, which need freshness, and which need both.
• Anything that needs private content or per-user permissions needs an owned index.
• Anything that needs the open web or same-day freshness needs search, or a crawl you can afford to run.
• Pick a hybrid pattern, and log the route and the retrieved passages for every answer.
• Write down the allowed sources for each question type.
• Model the costs of both options at your expected volumes.
• Build a test set from real questions and check retrieved passages by hand before you choose.
• Make "no relevant source" a supported answer in the interface.