Computer-use and browser agents
Computer-use and browser agents operate software the way a person does: they look at screenshots or the page structure, decide where to click or what to type, act, and look again. They can work with applications that have no API, which makes them useful for legacy systems, websites and one-off tasks. They are also slower, more expensive and more fragile than API-based tools, and every web page they read can contain instructions aimed at them, so use an API when one exists, run them in isolated environments, and require approval for anything that submits, buys or sends.
What they are
Most agents use tools you build: functions that call APIs and return data. A computer-use agent gets a different kind of tool: a screen.
It receives a screenshot (or, for browser agents, often the page's structure as text), and it can take actions like moving the mouse, clicking, typing, scrolling and pressing keys. A multimodal model reads the screen, decides the next action, the system performs it, takes a new screenshot, and the loop continues.
Anthropic released computer use for Claude in October 2024, and OpenAI released its Operator agent in January 2025, followed by browser and computer-use capabilities from other providers and open-source projects.
Browser agents and desktop agents
- Browser agents work inside a web browser. They can use the page's structure (the DOM or accessibility tree) as well as screenshots, which makes them more accurate. Frameworks like Playwright, which developers use for browser testing, often power them.
- Desktop agents control a whole operating system: any application, file manager or settings screen, usually through screenshots alone.
Where they are useful
- Systems with no API: legacy internal tools, government portals, supplier websites.
- Tasks across many websites, like gathering information that isn't available in one source.
- Testing web applications by exploring them like a user.
- One-off or rare tasks where building an integration isn't worth it.
- Helping a person with their own browser, with them watching.
Where they fall short
- Slow: every step needs a screenshot and a model call, so tasks that take a person a minute can take the agent several.
- Expensive: screenshots are many tokens each, and long tasks need many of them.
- Fragile: a pop-up, a layout change, a slow-loading page or a CAPTCHA can break a run.
- Less reliable than APIs: misreading the screen or clicking the wrong element is common, especially on complex pages.
If an API or a structured export exists, use it. A tool that calls an API returns exact data in milliseconds, while a browser agent reads it off a screen.
Security risks
Computer-use agents face the hardest version of prompt injection. Every web page they visit is untrusted content that goes straight into the model's context. A page can contain text, visible or hidden, telling the agent to ignore its task, visit another site, download a file or enter data somewhere.
Combine that with access to a logged-in browser, saved passwords or a company desktop, and the risk is real.
Protections:
- Run the agent in an isolated environment, a virtual machine or container, without access to your main accounts and files.
- Give it only the logins it needs for the task.
- Restrict which websites it can visit where possible.
- Require approval before it submits forms, makes purchases, sends messages, changes settings or enters sensitive information.
- Watch or record its sessions, and review what it did.
- Never let it handle passwords, payment details or one-time codes on its own.
Building with them
If you build a computer-use or browser agent:
- Give it a narrow, well-defined task and a clear way to know when it's done.
- Prefer browser automation with page structure over pure screenshots when you control the environment.
- Add step limits, timeouts and checkpoints, as for any agent.
- Measure success rates on your real tasks, because demos are much easier than real websites.
From the floor
I haven't built a computer-use agent in a session yet, so this page is written from provider documentation and research. The closest thing we did is the coding agent work, where Claude Code and our own DevFix agent run commands and edit files, and the same rules apply: narrow tasks, an isolated environment, and a person checking what changed.