Free and self-hosted web search for agents: what it really costs

Free web search is a sensible place to start. A prototype agent, a personal assistant on your own machine or an internal tool with a handful of users does not need a contract. But free options are free in different ways. Some move the cost to your time, some to the privacy of your queries, and some to legal risk you have not read about yet. This guide sorts the options, lists the costs that never appear on a price page, and gives you concrete signals for when to move to a paid service.

What free can mean

Official source APIs are the most overlooked row. For a narrow question, asking the authority directly often beats general web search on both accuracy and terms. The guide to web search for local AI agents covers wiring such sources into an agent that runs on your own machine.

The costs that are not on a price page

• Setup and upkeep. A self-hosted instance needs a server, updates, a decision about which engines to query, and attention whenever an upstream engine changes and results quietly disappear.
• Blocking. Public engines notice automated traffic. A metasearch instance serving an agent that fires many queries can be throttled or challenged, and the symptom is often empty or thinner results rather than a clear error.
• Thin results. Metasearch generally returns titles, links and snippets. To ground an answer you will fetch pages yourself, which brings its own blocking, parsing and politeness questions.
• Monitoring. Nobody tells you when a free source degrades. You need your own checks: a few canary queries with known answers, run on a schedule, alerting when results change shape or vanish.
• Your time. Count the engineering hours honestly. Free search that eats a day every month is not free.

Read the terms before you write the integration

Free tiers and public engines come with terms, and agents push against them in ways a person typing queries never does. Check:

• Is commercial use allowed, or only personal and evaluation use?
• May you store results, and for how long? May you show them to users, and with what attribution?
• May results, or text derived from them, be used to train or tune a model?
• Does your metasearch setup send queries to engines whose terms forbid automated access? Open-source software does not change the upstream terms, and the risk stays with whoever runs the instance.
• For official APIs: is there a required user agent, a contact address or a stated request etiquette? Follow it, or expect to be blocked.

How free options fail

Paid services fail too, but usually loudly: an error code, a status page, a support channel. Free options tend to fail silently. A blocked engine returns nothing, a changed page layout breaks a parser, a free tier hits its limit and your wrapper swallows the error. To an agent, all of these can look like a search that found nothing, and the agent will report with confidence that the information does not exist.

Defend against that in code. Distinguish four outcomes: results, a genuine empty result, a provider error and a timeout. Surface the last two as unavailability, never as absence. Record which engine or source produced each result, so a sudden drop from one of them is visible. The guide to web search in a self-hosted personal agent applies the same discipline to a personal setup.

Your queries say more than you think

An agent's search queries are rewritten from its task, so they carry the task's context: a customer's name, a symptom, a deal under consideration. Sending them to a public metasearch instance that someone else runs hands that operator a log of your work. If you use metasearch at all, run your own instance and keep it off the public internet. Keep sensitive work away from free tiers whose terms let the provider retain or reuse queries.

When to graduate to a paid service

Any one of these signals is a good reason to move:

• People outside your team now depend on the answers.
• You need a written commitment on availability, notice before changes, and someone to call.
• A legal reading of the free terms finds storage, display or training restrictions you cannot work within.
• You need page content or query-focused passages rather than links, and fetching pages yourself has become a project.
• Keeping the free setup running costs more engineering time than a subscription would cost money.
• You need steady quality in languages and regions your free sources cover poorly.

When you switch, compare candidates on your own queries with the scoring sheet in picking a web search API. Keep a thin abstraction layer, so the free option can stay on as a fallback for paths that are not critical.

Where GoCux fits

GoCux does not search the open web, free or paid, so it replaces none of the options above. What it changes is an earlier question: whether a given job needs web search at all. Many internal questions are answered by material you already hold. GoCux Knowledge search works across a project's own documents and returns passages with their citations. Web pages you gathered through an authorized browsing or export step come in with their original address and retrieval date, so their age stays visible. The note on bringing the evidence you already have describes that approach. When research does need outside sources, the permitted ones go into an allowlist that a person approves before the run, as the source policy explains.

A sensible starting setup

1. Start with official source APIs for the domains you care about and a provider's free tier for general queries, after reading both sets of terms.
2. If you self-host metasearch, run your own instance, choose the engines deliberately and keep it private.
3. Put every source behind one interface that returns results, empty, error or timeout, never a bare empty list.
4. Add canary queries with known answers and check them on a schedule.
5. Keep a short note of each source's terms next to the code that calls it.
6. Decide in advance which signal will trigger a move to a paid service, so the decision is not taken in the middle of an outage.