Picking a web search API for AI: a scoring sheet you run yourself

There is no single right web search API for AI work. There is the one that returns useful material for your queries, under terms you can live with, at a cost that still makes sense once an agent starts issuing several searches per task. Most published comparisons rank providers on someone else's queries. This guide gives you a method instead: what an AI application needs from search, the main ways providers obtain their results, what to check on freshness, terms and cost, and a scoring sheet you can run on your own query set.

What an AI application needs from search

A person scanning a results page fills gaps with judgment. A model reading results through a tool call cannot, so the API has to deliver more of the work up front. An AI application needs five things:

• Relevant URLs for queries as your users phrase them, including the long, conversational queries an agent tends to write.
• Text the model can use, not just a title and a two-line snippet. Thin snippets are a common cause of answers that sound grounded but are not.
• Metadata: publication or update date, retrieval date, language and domain, so the application can filter and cite properly.
• Controls: include or exclude domains, restrict by date, region and language.
• A stable, structured response that fits a tool-calling schema and does not change shape without notice.

There is also a budget dimension that classic search never had. Every result you pass to a model consumes context. Returning ten full pages for every query is expensive and often worse than returning a few passages that actually address the question. Agents make this sharper: one task may run many searches as the agent refines its query. If the term is new to you, start with what a web search API is.

Result types and what each is good for

Decide which result type your application consumes before comparing; otherwise you end up comparing one provider's snippet endpoint with another's content endpoint.

Own index, metasearch or SERP scraping

Where results come from shapes coverage, freshness, terms and long-term risk. Ask every provider which of these it is; a mix is common and fine, as long as you know.

• Own index. The provider crawls the web and builds its own web index. That gives it control over ranking and direct access to page content, but a large general index is costly to build and refresh, so few organizations run one. Brave Search API and Mojeek are examples of search APIs backed by an independent index.
• Metasearch. The service sends your query to several engines and merges the results. SearXNG, for example, is an open-source metasearch engine you can host yourself. Breadth is the appeal; dependence on upstream engines and their terms is the risk.
• SERP scraping. The service fetches a major search engine's results page and returns it as structured data. SerpApi and Serper are examples of this kind of service. Results resemble what people see in that engine; check whose terms apply and who carries the risk if they are enforced.
• Search designed for AI applications. A newer group, including Exa and Tavily, shapes its output for models: passages, page content, sometimes direct answers. Ask them the same sourcing question.

One more risk to price in: a search API you depend on can be withdrawn or restructured. Keep a thin abstraction between your application and any single provider. Replacing a search API you depend on walks through that situation.

Freshness has two clocks

"Fresh results" means two separate things. One clock is when the page itself changed. The other is when the provider last fetched it. A page updated this morning is stale to your application if the index last saw it a month ago.

Ask three questions. Does the API return a publication date, a retrieval date, or both? Can you filter or sort by date? And how quickly do new pages appear for the kinds of queries you care about? News, pricing pages and documentation all behave differently.

You can test the last one directly. Collect a handful of queries whose correct answer only recently appeared on the web: a page your own team published, a release note, a news item with a known time. Run them against each candidate on consecutive days and note when each one first returns the new page.

Terms and cost model

Terms of use are usually read last and matter most. Before you build an integration, check:

• May you store results, and for how long? May you show them to your users, and with what attribution?
• May results, or text derived from them, be used to train or fine-tune a model?
• Are your queries logged, retained or used by the provider? Queries written by an agent often contain customer names and internal context.
• Where are queries processed, and is a data processing agreement available if you need one?
• How much notice will you get before the API changes or ends?

On cost, compare the shape of the pricing model, not the headline number. Providers charge per query, per result, per page fetched, per feature tier or through prepaid credits, and the same workload can cost very differently under each. For an AI application, the unit that matters is cost per completed task: if your agent typically needs several searches and a few page fetches to finish one, price that combination.

A scoring sheet you run yourself

Vendor benchmarks answer the vendor's questions. Your own evaluation takes a few days and answers yours. The procedure:

1. Build a query set from real traffic. Take 30 to 50 queries from logs, support tickets or agent traces. Mix navigational, factual, recent, long-tail and non-English queries, plus the jargon of your domain.
2. Write down what a good result looks like for each query before you run anything. Include a few queries where the honest answer is that nothing relevant exists.
3. Run every candidate with the same parameters and store the raw responses, so you can re-grade later.
4. Grade blind. Hide provider names from whoever grades relevance.
5. Repeat the run on another day. Results move, and a single run hides that.

Set the weights before you see any results, and keep the empty-answer queries in the set: a provider that confidently answers a question with no good answer will pass that confidence on to your application. The guide to benchmarking web search on your own queries goes deeper on grading. For the reviewer's side, see evaluating search in the docs.

Where GoCux fits

GoCux is not a web search API and does not read the open web, so it does not belong on your scoring sheet as a candidate. Its Knowledge search answers questions across one project's own documents and returns passages with their citations. It treats an empty result as a legitimate answer and keeps "the service was unavailable" distinct from "nothing was found", which is the same discipline this guide asks of any search provider.

If your agent uses a web search API, GoCux is relevant downstream of it. A research plan names the question, the ordered steps and a source allowlist, and it has to be approved before it runs; execution cannot stretch it, and an allowlist left empty permits no source at all. Findings backed by a quotation stay separate from questions that no source answered. The source policy page explains the allowlist, and the comparison page offers a repeatable way to weigh tools against the task at hand.

What to verify before you commit

• You know whether each candidate runs its own index, metasearches, scrapes results pages, or mixes these.
• You compared the same result type across providers.
• Your query set came from real traffic, was graded blind and was run more than once.
• Dates come back in responses, and you tested how quickly new pages appear.
• The terms allow your storage, display and any training use, and you know what happens to your queries.
• You priced a completed task, not a single query.
• A thin abstraction layer lets you switch providers without rewriting the agent.
• Your code handles an empty result and an unavailable service differently.