Recall and accuracy in agent search: how to measure them yourself

Ask which search API has the best recall and accuracy and you will get confident answers, mostly from the vendors themselves. The honest answer is that the question has no general solution. Recall and accuracy are properties of a search system on a particular set of questions, judged against a particular idea of what counts as relevant. Change the questions or the judge and the order changes. This article explains what the two words mean when an agent, not a person, is doing the searching, why published figures are hard to line up against each other, and how to measure both on the work you actually need done.

Recall, precision and accuracy are different questions

Three terms get blurred in marketing copy. Keeping them apart is most of the work.

• Recall asks: of all the relevant material that exists, how much did the system return? If forty documents answer your question and the search surfaces thirty of them, recall is thirty out of forty. It is usually measured at a cutoff, such as the top ten results, because nobody reads an unlimited list.
• Precision asks the reverse: of what came back, how much was relevant? Ten results with seven useful ones is seven out of ten at that cutoff.
• Accuracy usually means something else in AI search: was the final answer correct? Many AI search products return a written answer rather than a list, and their published figures grade that answer, not the retrieval behind it.

Two more properties matter for agents. Faithfulness: does each claim in the answer actually appear in the source it cites? And abstention: when nothing relevant exists, does the system say so, or does it produce something plausible? A system can score well on answer accuracy while citing pages that do not support its sentences, and it can look precise while quietly missing the one document that changes the conclusion.

Why recall matters more once an agent is searching

A person who searches reads the results page, notices what is missing and tries again. An agent can do that too, but only if the material it needs appears somewhere in what it sees. Low recall fails silently: the agent writes a fluent summary of the sources it found and never learns that a more important one existed. In due diligence or list building, the missing item is often the expensive one.

Agents also change where recall is measured. One query may return a narrow set, but an agent that reformulates five times may still reach the relevant document. So there are two levels worth measuring: query-level recall, what one search returns, and task-level recall, what the whole search loop collects before the agent stops. A provider that looks weaker per query can do fine at task level if its results help the agent rephrase, and a strong per-query provider can be wasted by an agent that stops too early. The loop itself is described in agentic search.

Precision has its own cost in agent work. Every irrelevant passage takes up context and gives the model something to misquote. High recall with poor precision means a longer, more expensive and noisier context, so the useful target is usually good recall within a small cutoff, not maximum recall at any length.

Why vendor figures are hard to compare

Published numbers are not necessarily wrong; they answer a narrower question than the headline suggests. Before comparing two, check:

• Which questions. Public benchmarks differ a lot. SimpleQA, for example, is a set of short factual questions with single correct answers; BrowseComp targets facts that are deliberately hard to find; retrieval collections such as BEIR measure document ranking across many domains. A system tuned for one tells you little about another, and none of them is your workload.
• What level is graded. Some figures grade the retrieved documents, some grade a generated answer, some grade a full agent with its own reasoning on top. These are not the same measurement.
• Who judged relevance. Human annotators, a model acting as judge, or an answer key. Model judges are convenient and have their own biases, including leniency towards fluent answers.
• How recall was defined. True recall needs the full set of relevant documents, which nobody has for the open web. Evaluations approximate it, often by pooling: they collect results from several systems and judge the combined set. A system that finds documents none of the others found can be under-credited.
• Cutoff, settings and date. Top five or top twenty, with or without full page content, with which filters, run when. Web results move, so a run from months ago describes a different index.
• Who ran it. An evaluation run by the vendor, on a set the vendor chose, with settings the vendor tuned, is a starting hypothesis rather than a result. Check whether the queries, judgements and raw outputs are published so anyone can repeat the run.

What “most powerful” should mean for your application

Lists of the most powerful search API for AI usually collapse many properties into one ranking. Power is not one axis. For an agent it is a bundle:

The most capable tool for a breaking-news assistant can be a poor choice for exhaustive company lists, and the reverse. Weight these dimensions for your task before reading anyone's ranking. If latency is the binding constraint, see search for latency-sensitive applications; for the wider selection process, see choosing a search tool for AI agents.

How to measure recall and accuracy on your own task

A useful evaluation fits into a few days and produces numbers you can defend, because you know exactly what they measure.

1. Pick the task, not the API. Write one sentence about what the agent must produce, for example: “a list of every regulator notice from the last quarter that mentions our product category, with the source page for each.”
2. Build a gold set. Take a few dozen real questions from that task. For each, record the documents or facts a careful person would expect to find. Where you cannot know the full set, pool: run every candidate, add what each one finds, have a person judge the combined list, and note that your recall is relative to that pool.
3. Include questions with no good answer. They measure abstention, which is otherwise invisible.
4. Measure at two levels. Query-level recall and precision at the cutoff your agent actually reads, and task-level recall after running your real agent loop with each provider plugged in.
5. Grade answers separately from retrieval. Mark each final answer correct, partly correct or wrong, then check each citation: does the cited passage say what the sentence claims?
6. Grade blind and repeat. Hide provider names from graders and rerun on another day to see how far results drift.

A sample acceptance rule you could adopt: “For every gold-set question, each document marked high importance appears in the first ten results of at least one query the agent issues; every factual sentence in the answer is supported by a cited passage; questions with no good answer end in an explicit ‘not found’.” The grading mechanics, step by step, are in evaluating web search on your own queries.

Making coverage visible in the output

Measuring before you buy is half the job. The other half is making recall gaps visible in every result after you go live, because no provider's recall holds on every question. Three habits help. Keep questions the search could not answer on a separate list instead of dropping them, as argued in missing evidence is an open question. Report what was searched and what was not. And read a result count as a description of what was found, never as proof that the set is complete.

That second half is where GoCux is useful. GoCux is not a search API and does not read the open web, so it is not a candidate on your shortlist. Its FindAll takes matching criteria and columns and returns a table in which each row carries its own evidence, and it states coverage limits instead of smoothing them over; the count it returns is not offered as proof that nothing was missed. In research, findings that rest on a quotation are kept apart from questions where no source turned up, and a citation is treated as a pointer for a reviewer to check rather than as proof. The benchmarks page describes evaluating the work itself, not only the answer.

What to verify before you trust a recall or accuracy claim

• You know whether the figure grades retrieval, a generated answer or a whole agent.
• The question set, the judge and the cutoff are stated, and they resemble your task.
• The run can be repeated: queries, judgements and raw outputs are available.
• You measured recall at the cutoff your agent actually reads, at query level and at task level.
• Your gold set includes questions with no good answer, and you scored abstention.
• Citations were checked for faithfulness, not just presence.
• Your production output lists unanswered questions and coverage limits instead of hiding them.