Search under a latency budget: speed, trade-offs and fallbacks

Most search APIs are quick enough for a report an agent writes in the background. They are not automatically quick enough for a voice assistant that must answer before the caller starts talking again, or for a search box that suggests results while someone types. Once speed becomes a product requirement, the selection question changes. It is no longer which provider returns the best results, but which one returns good-enough results inside the time you can afford, and what you do when it does not. This guide covers where latency really matters, what you give up to get it, how caching and fallbacks fit in, and how to measure on your own traffic instead of trusting a published figure.

Where speed changes the product

• Voice. Silence on a call reads as failure. Search sits between speech recognition and speech synthesis, so its delay adds straight to the pause the caller hears. Many voice designs play a short acknowledgement while the search runs; that buys a little time, not unlimited time.
• Chat. People tolerate a brief wait before the first words appear, then expect a steady stream. If search runs before generation, its delay is the entire wait before the first word.
• Autocomplete and instant results. Requests fire on keystrokes, and a response that arrives after the next keystroke is wasted. Here you need cancellation as much as raw speed.
• Agent loops. An agent that searches, reads and searches again pays the delay at every step. A modest delay per call turns into a long task when the calls are sequential and numerous.

Speed matters much less for overnight research, scheduled monitoring and long-form reports, where depth and coverage should win. Do not let a latency-driven choice for one feature become the default for everything. For those other jobs the general guide to picking a web search API is the better frame, and choosing a search tool for AI agents covers the agent side.

Write the latency budget down

Before comparing providers, split the total time a user will wait into parts. Search gets a share of it, not all of it. A template you can copy:

Fill the budget column from the experience you want and the measured columns from your own tests. If the search line on its own already exceeds its budget at the slow tail, no tuning elsewhere will rescue it. Remember that people remember the slow tail, not the typical case.

What a faster response usually costs

Low latency is rarely free. Configurations that answer quickly often do so by doing less work per query. The usual trade-offs:

• Snippets instead of content. Titles and short excerpts come back faster than fetched, cleaned pages, but thin snippets give a model less to ground its answer on.
• Fewer results, less reranking. Skipping a second ranking pass saves time and costs relevance on harder queries.
• Precomputed or smaller indexes. Serving from a prepared index is quick; it may also be older, or thinner on long-tail and non-English queries.
• Less query understanding. Fast modes may skip the rewriting or expansion that helps with long conversational questions.

Some providers offer several modes or depths behind one endpoint. That is genuinely useful: a quick mode on the interactive path, a deeper one in the background. Check what each mode returns, not just what it is called, and run both on the same queries. The guide to judging recall and accuracy of AI search helps you see what a quick mode leaves out.

Caching without serving stale answers

The fastest search call is the one you never make. Caching pays off when queries repeat, and in support assistants, autocomplete and product search they repeat more than people expect.

• Normalize before you build the key. Lowercase, trim, collapse whitespace and order filters consistently so equivalent queries share one entry.
• Set expiry by query class. Reference and documentation questions can live longer; prices, news, availability and anything containing today should expire quickly or skip the cache.
• Treat similarity caching with suspicion. Reusing results for a query that merely looks similar is fast and sometimes wrong: two questions that differ by one company name are close in wording and far apart in meaning.
• Check the terms. Some providers limit how long results may be stored. Your cache policy has to fit the contract.
• Prefetch where intent is predictable. When a user opens a company profile, the obvious follow-up searches can start before they ask.

Timeouts and fallbacks

A latency target is only real if something defined happens when it is missed. Decide that behaviour in advance:

1. Set a hard timeout for each call, derived from your budget, and cancel the request when it fires.
2. Hedge with care. Sending a second request to a backup provider when the first is slow can trim the slow tail, at the price of paying twice for some queries. Keep it to the interactive path.
3. Degrade openly. Fall back to a cached result with its age shown, to a narrower source, or to an answer that says current information could not be checked. Never present an answer made without search as if it were grounded.
4. Keep timeouts apart from empty results. The search did not finish and the search found nothing call for different responses, and they belong in different log lines.

That last point matters more than it looks. An agent that reads a timeout as an empty result will confidently report that something does not exist.

Measuring speed on your own queries

• Take real queries from your logs and group them by class: short navigational, conversational, long-tail, non-English.
• Measure from the servers and region where your application actually runs, not from a laptop.
• Record typical and slow-tail times, for example the median and the 95th or 99th percentile, separately for first-time and repeated queries.
• Measure end to end: search plus any fetch and rerank, up to the moment generation can begin.
• Count timeouts and errors as failures, never as missing data points.
• Grade relevance on the same run. A fast wrong answer is still a wrong answer.
• Repeat at different times of day, because load on the provider varies.

The method in evaluating web search on your own queries covers relevance grading; add the timing columns from this page to it.

Where GoCux fits

GoCux is not a search API and does not read the open web, so it is not a candidate for your interactive search path. It is relevant around that path in two ways. The first is the distinction above. GoCux Knowledge search, which works over a project's own documents, treats an empty result as a valid answer and reports unavailable as something different from nothing found. Its reference on warnings and errors likewise treats an unavailable dependency as its own kind of error with its own remedy. That is the behaviour your fallback code should copy.

The second is what happens after a fast answer. A voice or chat assistant can look things up as quickly as you like, but a step that changes something, such as sending a message or scheduling work, should not ride the same fast path. In a GoCux conversation, a tool call that would change anything is held until a person decides, so reading and acting deliberately run at different speeds. The note on why an agent should ask before it acts explains the reasoning.

Checklist before you choose

• You know which features are latency-sensitive and which are not.
• A written budget gives search a defined share of the total wait.
• You measured typical and slow-tail times from your own servers, per query class.
• Relevance was graded on the same runs as the timings.
• The cache expires by query class and fits the provider's storage terms.
• Every call has a timeout, and you know what the user sees when it fires.
• Timeouts, errors and empty results are logged and handled separately.
• Consequential actions do not share the fast path with lookups.