Testing a search API on your own queries, from gold set to repeat runs

Every search provider can show you a chart where it wins. None of those charts used your queries, your agent or your definition of a good result. If search feeds an AI application, the only evaluation that predicts how it will behave in production is one built on the questions your system actually asks. This guide sets out the full method: deciding what results are for, building a gold set, scoring relevance and evidential support separately, dealing with recall when the denominator is unknown, and making the whole thing repeatable enough to rerun next quarter. At the end there is a cut-down version you can run in an afternoon.

Decide what the results are for

A person scanning results tolerates a lot: they skip the spam, open two tabs and work out the answer themselves. A model reading results does less of that. It tends to trust what is in front of it, it may only see snippets, and it cannot tell you that the page it needed was missing. So evaluate against the job the results do in your system, written as one sentence:

For each user question, the top results should include at least one passage from an acceptable source that is enough, on its own, to answer correctly.

That sentence already implies most of the metrics: whether the right material was found, whether it actually supports the answer, whether the source is acceptable, and whether it arrived in a form the model could read. It also tells you to evaluate at two levels. Retrieval evaluation scores the result list directly. End-to-end evaluation scores the final answer your agent produces with each search option plugged in. You need both, because a better result list does not always produce a better answer.

Build a gold set from real queries

Start from what your system has really been asked: production logs, agent traces, support tickets, the questions your team types during testing. Synthetic queries written by the evaluator tend to be cleaner and easier than real ones, which flatters every provider.

Sort the queries into types, because providers differ by type and an average hides it:

For each query, record what a correct answer looks like and what makes a source acceptable, before you look at any provider's results. Writing the judging criteria after seeing results quietly tunes them to whichever provider you saw first.

{
  "id": "q-0412",
  "query": "minimum notice period for ending a commercial lease in ...",
  "type": "long_tail",
  "answer_must_contain": ["the notice period", "the legal basis"],
  "acceptable_sources": "official legislation site or a court; law firm summaries are secondary",
  "time_sensitive": false,
  "expires": null,
  "notes": "two answers circulate; the statute in force decides"
}

Size the set by type rather than in total. A modest number of queries per type, judged carefully, tells you more than a large pile judged carelessly. Grow it over time by adding every production failure you investigate.

Score relevance and support separately

Topical relevance asks whether a result is about the question. Support asks whether it contains the specific passage that justifies a correct answer. For AI use, support is what matters, and the gap between the two is where many search evaluations go wrong: a result list full of on-topic pages that never state the fact scores well on relevance and still leaves your agent guessing.

A simple graded scale for each result:

• 3, supports: contains a passage that answers the question, from an acceptable source.
• 2, relevant: on topic and useful as context, but does not settle the answer.
• 1, marginal: mentions the subject in passing.
• 0, irrelevant or wrong: off topic, outdated, or states an incorrect answer.

Record source acceptability and freshness as separate flags rather than folding them into the grade, so you can later ask questions such as how often the only supporting result came from an aggregator. Note duplicates too: five copies of the same press release are one result, not five.

Use two human judges on at least a sample, with written guidelines, and settle disagreements by discussion; disagreements usually expose an ambiguous guideline. A model can act as a judge to scale this up, but only after you have compared its labels with human labels on the same items and found them consistent, and you should keep spot-checking afterwards. Hide which provider produced each result while judging.

Precision, recall and the unknown denominator

Precision is easy: of the top k results, how many are supporting or relevant. Choose k to match how many results your agent actually reads; scoring the top twenty when the agent reads five measures something your system never sees.

Recall is hard, because on the open web nobody knows how many supporting pages exist. The standard workaround, borrowed from information retrieval research such as TREC, is pooling: take the top k results from every candidate, add what a person finds by searching manually, judge the whole pool, and treat the supporting results in it as the known set. Recall is then measured against the pool, not against the web. Two consequences follow. A provider that finds good pages nobody else found will look worse until those pages are judged and added, so judge new results whenever you add a candidate. And recall figures from different pools cannot be compared.

For agents, a more useful recall question is usually per query rather than per document: did the top k contain at least one supporting result? An agent typically needs one good passage, not all of them. Ranking metrics such as nDCG matter when order matters, for example when the agent reads only the first few results.

Score the unanswerable queries separately. The right behavior there is to return little or nothing convincing; a provider that always returns a confident-looking page will look generous on answerable queries and dangerous on these. Errors and timeouts deserve their own column as well. A request that failed is not a request that found nothing, and counting failures as empty results hides reliability problems.

Make it repeatable

Search results drift. Indexes update, ranking changes, and some providers vary results between identical calls. An evaluation you cannot rerun is an anecdote. To make yours repeatable:

• Store the raw response for every call, with a timestamp and the exact parameters used.
• Run each query more than once, on different days, and report the spread as well as the average.
• Version the gold set and the judging guidelines; a change to either starts a new series.
• Re-judge time-sensitive queries on every run, and retire queries whose answers have expired.
• Use one harness for every candidate, with the same timeout, result count and content settings.
• Keep cost and latency as separate columns measured in your own environment, not copied from a vendor page.

Then run the end-to-end comparison: the same agent, prompt and model, with only the search backend swapped, graded on answer correctness and on whether each cited source really supports the claim it is attached to. When an answer is wrong, attribute it: retrieval missed, ranking buried the right page, content extraction failed, or the model misread a good source. Only the first two are the search provider's fault. This is the idea of evaluating the work, not just the answer, applied to search; citations that hold up covers what to check on the citation side.

The same method applies to search over private documents. GoCux is not a web search API; its Knowledge search answers questions inside one project's documents and returns passages with their citations. An empty result there is a legitimate answer, and “unavailable” is reported differently from “nothing found”, which is exactly the distinction the unanswerable and error columns above are designed to test. The guide to evaluating search in the docs applies this to project knowledge.

The afternoon version, and what to verify

If you need a first answer this week rather than a full program, a reduced version still beats a vendor chart:

1. Pull a few dozen real queries from your logs, covering at least four of the types above, including some unanswerable ones.
2. Write the acceptable answer and source for each before running anything.
3. Run every candidate with identical settings and save the raw responses.
4. Judge the top results blind on the 0 to 3 scale, flagging stale and secondary sources.
5. Report per type: the share of queries with at least one supporting result, the behavior on unanswerable queries, and failures.
6. Rerun the same set a few days later and see whether the conclusion holds.

Before acting on any result, check that the queries were real, that judging was blind, that k matches what your agent reads, that failures were counted as failures, and that the conclusion held on a second run. To choose among providers once you have figures of your own, see how to choose a web search API and judging recall and accuracy.