How an agent system fits together and where control belongs

Agent architecture diagrams usually put a model in the middle and draw arrows to everything else. That picture hides most of the engineering. The model is the part you are least likely to build yourself and the most likely to swap out. Whether an agent system is dependable is decided by the software around it: how the loop runs, how tools are exposed, what is remembered, where evidence is kept, where a person approves and what gets recorded. This page walks through those components, the handful of patterns most systems are built from, where web access plugs in, and the decision that matters most: where control sits.

Six components of an agent system

How the parts connect matters as much as the parts themselves:

request     -> loop: plan, act, observe
loop        -> policy check -> (approval gate if consequential) -> tool executor
tool        -> evidence store (passage, source, retrieved at)
loop        -> memory (proposals, scoped)
every step  -> audit trail (append-only)

Inside the loop

Each turn of the loop assembles context (the goal, constraints, relevant memory and what has happened so far), asks the model for the next action, validates that action against a schema, executes it through the tool layer and records the observation. The details that make the loop reliable sit outside the model:

• Stopping conditions. The goal is met, the agent is stuck and says so, a budget is reached, or the next step needs a decision. Each should be a distinct, visible outcome.
• Budgets. Wall-clock time, tool-call count and spend are counted and enforced by the runtime, not requested politely in the prompt.
• Structured actions. Tool calls are parsed into typed arguments and rejected if malformed, rather than executed on a best guess.
• Durability. Long work survives restarts by persisting state between steps. If durable execution is not available, the system should say so instead of pretending to make progress.

Tools and the policy boundary

Every tool should declare three things besides its schema: what kind of side effect it has (read, write, send, delete, export, schedule), what credential scope it needs, and which resources it may touch. With those declarations in place, a policy check can sit between the loop and the executor and decide each call on its own terms:

def execute(call, run):
    # every refuse() is also written to the audit trail
    kind = tools[call.name].effect          # read, write, send, delete, export...
    if not run.scope.permits(call):
        return refuse(call, reason='outside scope')
    if kind in CONSEQUENTIAL:
        decision = gate.ask(run, quote=call) # reviewer sees the exact call
        if not decision.covers(call):        # checked again at execution time
            return refuse(call, reason=decision.reason)
    audit.append(run, call, kind)
    creds = vault.reference(run.scope)      # server-side, never in the prompt
    return tools[call.name].run(call.args, creds)

Three properties of that sketch are worth copying. The check happens in code, so no instruction in the context can talk its way past it. The reviewer sees the exact call that was held, not a paraphrase of it. And a decision covers one specific call; the system does not remember “approved” as a general mood. Protocols such as MCP standardise how tools are described and invoked, which makes connecting them easy. They do not decide whether a given task may use a given tool. That is the policy layer's job.

Memory and evidence are different stores

Teams often keep everything an agent learns in one place. Split it into three:

• Working context lives for one run and is discarded or summarised at the end.
• Long-term memory holds facts and preferences across runs. Each entry should have a scope (session, user, project or company), a source, a reason and a status of proposed or approved. Corrections create a new version. The memory documentation describes this model in more detail.
• Evidence is what the agent retrieved: passages with their source reference and retrieval time, kept unchanged. Findings point into this store.

When memory and evidence mix, an agent's earlier summary starts being cited as if it were a source, and nobody can tell which claims were ever checked. Keeping them apart is cheap at design time and very hard to retrofit.

Common patterns

Start with the single agent and move to another pattern only when you can name the problem it solves. Each extra agent adds context handoffs, and every handoff is a place where scope and evidence can get lost.

Where web access fits

“Web access” covers four different capabilities, and they belong at different levels of risk:

• Search. A query goes in and ranked results come out, typically through a web search API. Read-only, but the results are untrusted text.
• Fetch and extract. An address goes in and page content or structured fields come out. Also read-only, also untrusted.
• Browser automation. A browser agent clicks, types and submits forms. Because it can act in a live session, its consequential steps need the gate.
• Crawling. Systematic collection across many pages, usually a separate pipeline with its own rules on robots files, request rate and site terms.

Model all four as tools behind the same policy check, and add a few web-specific rules. Give each task a source allowlist, and treat an empty list as permitting nothing. Store what was retrieved in the evidence store, with its address and retrieval time, before the agent reasons over it. Never let page content change the agent's instructions or permissions; a page that says “ignore previous instructions” is data. And treat an address as a location, not as permission: getting past a login needs an explicit grant.

Where control must sit

The principle running through every section is that the controls which matter live in the harness, outside the model, at the boundary where actions happen. A practical test is to imagine deleting the system prompt. Any safeguard that disappears with it was never a safeguard.

• Scope, budgets and the stop control are enforced by the runtime.
• Credentials are server-side references the model never sees.
• Approvals are bound to one attempt and re-checked when it executes.
• The audit trail is append-only and includes refusals and rejected decisions; what an agent audit trail should record goes through the fields.
• Organisation boundaries are hard: a decision in one organisation authorises nothing in another.

The explainer on the agent harness looks at that surrounding layer as a whole.

GoCux as a worked example

GoCux places these controls in a shared workspace rather than inside each agent. Work runs as work orders with a goal, a scope and acceptance criteria. Seven consequential action classes pause for someone on the team; the held call is shown exactly as the agent made it, and a decision covers only that attempt and is checked again when it executes. Credentials stay on the server as references. Requests, decisions, tool runs and outcomes form one trail. Spend draws on visible credits under a daily cap, and one stop control halts new work. Tools connected over MCP run in your own compatible client, and configuring a server does not by itself authorise every task to use it. Governance covers the gate and the trail, and the seven action classes are explained one by one in a separate note.

Architecture review checklist

• Can the loop stop for each reason (done, stuck, limit, decision), and is each stop recorded distinctly?
• Does every tool declare its side-effect class, credential scope and resources?
• Is the policy check written in code, between the loop and the executor?
• Do consequential calls pause with the exact call shown, and is approval re-checked at execution?
• Are working context, memory and evidence separate stores, with memory entries scoped and versioned?
• Does every web tool sit behind a source allowlist, with retrieved content stored before use?
• Can page content ever change instructions or permissions? It should not.
• Does the audit trail record refusals and rejections, not only successes?
• Is there one stop control, and is resuming a deliberate act with a named person behind it?