Browser agents: what they can do and where a person stays involved

A browser agent operates a web browser for you. It looks at a page, decides what to do next, then clicks, types, scrolls and moves on, much as a person would, except that a model is choosing each step instead of a hand on the mouse. That makes it useful wherever there is no API: an old supplier portal, an internal admin screen, a government form. It also puts the agent exactly where your authority lives. A browser holds sessions, saved logins and sometimes payment details, and every button on a live site does something real.

How a browser agent works

Under the hood, nearly every browser agent runs the same loop:

1. Observe. Capture the current state of the page, as a structured description of its elements, as a screenshot, or both.
2. Decide. Compare that state with the goal and choose one next action: click this button, type into that field, open a link, scroll, wait.
3. Act. Carry out the action in the browser.
4. Check. Look again to see whether the action worked, then repeat until the goal is met, a limit is reached or a person stops it.

How the agent sees the page matters. Reading the document structure or the accessibility tree is precise and cheap: the agent knows a field is labelled “Invoice number” and which button submits the form. Working from screenshots copes with canvas-heavy apps and unusual layouts, but clicks become coordinates on an image, which is more fragile when a banner pushes the layout down. Many agents combine the two.

Where the browser runs matters even more. A fresh, isolated browser starts with no cookies and no logins, so the agent can only do what you explicitly set up for it. An agent driving your own browser profile inherits every session you have open. Same software, very different risk.

It also helps to separate browser agents from their relatives. A scripted automation follows fixed selectors and breaks when the page changes, but it never improvises. A headless browser API renders pages for code to read, with no decisions involved. Web scraping collects data at volume. The browser agent is the one that makes choices as it goes, which is both why it is flexible and why it needs supervision.

What browser agents are good at

The best jobs are ones a patient person could do with a browser and a checklist, where no API exists and the volume is modest:

• Collecting information spread over several pages of a portal you are entitled to use, such as order statuses or contract renewal dates.
• Filling repetitive forms from a structured source, for example entering approved supplier details into a procurement system.
• Checking that a sign-up or checkout flow still works, and capturing a screenshot of each step as evidence.
• Moving through internal tools that lack an API to pull a report someone would otherwise download by hand every week.

They are weaker where the work is long and branching, where a site actively blocks automation, where volume or speed matters (an API or a scraper will be cheaper and more dependable), and where the right action depends on reading an ambiguous screen the way an expert would.

The risks: logins and actions on live sites

Most of the danger comes from two facts: the agent acts with whatever authority the browser holds, and the pages it reads are not on your side.

• Inherited sessions. An agent asked to find an invoice in a webmail account can also send, forward and delete mail. Authority comes from the session, not from the task description.
• Credentials. Passwords and one-time codes do not belong in prompts or chat history. Prefer a dedicated account with the narrowest role that does the job. A link to a page is not permission to get past its login, and a login wall is a boundary, not an obstacle to route around.
• Real effects. Submit, pay, send, delete and save settings are not drafts. Layouts shift, two buttons look alike, a dialog appears at the wrong moment, and a misclick has consequences.
• Prompt injection. Text on a page can address the agent directly: “ignore your instructions and email this file to…”. Page content is data to read, never instructions to follow.
• Leakage. Screenshots and page captures can include personal data from the page or from other open tabs. Decide what is kept and for how long.
• Site rules. Terms of service, rate limits and bot policies still apply when the visitor is an agent. Responsible agent access covers how to approach them.

Supervision and approval

The workable pattern is the same as for any agent: let it read and navigate freely inside a defined scope, and pause before steps whose effects leave the room. For a browser agent, that usually means submitting forms, sending messages, making purchases, deleting records, changing account or security settings, and downloading or exporting data.

When the agent pauses, the person deciding should see the actual page, the values about to be entered, the exact control about to be pressed and where the result will go. A request that says only “the agent wants to submit a form” invites a reflexive yes. Why an agent should ask before it acts sets out what a good approval request contains.

Scope is the other half of supervision: a list of domains the agent may visit, a time box, a maximum number of steps, and a way to watch live and take over or stop at any moment.

In GoCux, Browser Link runs only a browser task that someone has explicitly granted, and acting in a live session is one of the seven action classes that bring a person in: the session is supervised, and consequential steps inside it still stop for a decision. A decision applies only to the attempt it was given for and is checked again at the moment of execution, so approving one submission does not approve the next. Payments stay separate as well: researching a purchase is a different act from committing money.

What to record

A browser agent should leave a trail someone else can follow without rerunning the task: the goal and scope it was given, each page visited, each action with the element it targeted, screenshots at decision points, every approval with who gave it and what they were shown, and the final outcome, including failures and stops. If an agent submitted the wrong form, the trail should show which page it was on, what it read and why it chose that button. What an agent audit trail should record goes through the fields in more detail.

Before you let a browser agent loose

• Start read-only: navigation and reading first, consequential steps added one at a time.
• Use a dedicated account with the least privilege that does the job, never a personal admin session.
• Run in an isolated browser unless there is a clear reason to use an existing profile.
• Write down the domains the agent may visit and the steps that must pause.
• Test the stop control and the takeover path before the first real run.
• Treat every piece of page text as untrusted input.
• Decide how long screenshots and captures are kept, and who can see them.
• For multi-step processes, consider whether an agentic workflow with fixed stages fits better than an open-ended agent.