What an AI agent actually does when it clicks for you

Ask an AI assistant to fill in a booking form and it looks like the AI is using your computer. It isn’t. The model never touches your screen, mouse or keyboard. It looks at pictures of the screen and sends back instructions, and other software does the clicking.

That split explains both what agents can do and where they go wrong.

It sees the screen as a photo

When an agent works in a browser or on a desktop, it starts with a screenshot. Google puts it simply: using screenshots, the model can “see” a computer screen.

The model studies the picture and decides what to do next. Then it replies with an instruction, not an action. Anthropic’s documentation lists the instructions Claude can send. They include clicking at a point on the screen, typing text and pressing a key or key combination. The point is given as screen coordinates, measured in the screenshot’s pixels.

Something else does the clicking

That instruction goes to a separate program running on the computer, in a browser or on a server. That program moves the pointer, clicks and types. Then it takes a new screenshot and sends it back to the model.

All three big model makers describe it the same way. Anthropic says Claude “doesn’t directly connect” to the computer, and the developer’s application “runs every call in an environment you control”. OpenAI tells developers: “You provide the environment and execute the model’s requests.” Google says developers must build the part that receives and carries out the actions.

The loop repeats, screenshot by screenshot, until the model says it’s finished. Each picture costs something. Anthropic says each screenshot uses roughly 1,000 to 1,800 tokens of input, the units AI companies use to measure and charge for text and images. Long agent jobs can get expensive.

The brain and the body

The Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC) has a neat way to put it: “If you think of the LLM as the brain, then the harness is the body.” The LLM, or large language model, is the AI itself. The harness is the software around it. ASD’s guidance says the model produces outputs but “does not directly control any resulting actions”.

The same idea covers agents that don’t use a screen at all. When an AI connects to your calendar or accounting app, it sends a structured request, a bit like a filled-in form. The software connecting the AI to that app runs it and reports back. Anthropic’s documentation says computer use is one of these tools too: the application still runs each call.

Why that matters: everything on screen is input

The model reads whatever is in the screenshot, so a web page can talk to it. Anthropic warns that instructions on web pages or in images “might override your instructions”. Google’s Gemini help pages warn of malicious instructions “hidden from you but visible to the AI agent”. This is called prompt injection.

ASD gives a business example. Attackers could hide instructions in phishing emails to convince an email-reading agent to download malware.

Agents also make plain mistakes. Anthropic says Claude “sometimes acts as if its actions succeeded” without checking, and Google calls Computer Use a preview capability that “may contain errors”. ASD has pointed to an ABC News report of an AI agent in a gym-booking system that removed another customer from a waiting list. Last month ASD’s ACSC also issued a high alert about AI agents acting without authorisation.

The brakes

Because a separate program carries out every action, it can also stop one.

• Confirmation before risky steps. Google says Gemini is designed to always ask you to review and confirm before certain actions, including purchases and submitting web forms. ChatGPT agent also pauses to confirm certain actions online, according to OpenAI’s notes.
• Checks on every action. In Google’s developer tools, a safety system can flag a proposed action as allowed, needing your confirmation, or blocked, and the developer’s app must enforce it.
• Approval modes. Claude in Chrome has a mode where Claude “pauses and asks for approval before each action”. Anthropic says some actions are off limits whatever you choose, including purchases and permanent deletions.
• Pausing when you look away. On sensitive sites such as email or banking, ChatGPT agent’s watch mode pauses when you become inactive or leave the conversation.

None of these makes an agent safe on its own; Anthropic says the chance of an attack is “still non-zero”. ASD’s advice to individuals is to restrict agents to low-risk, non-sensitive tasks and keep a human reviewing what they do.

For a business, that means giving an agent only the access the job needs. ASD warns against ever granting agents “broad or unrestricted access”, especially to sensitive data or critical systems.

Checklist

• Give the agent its own browser profile. Anthropic suggests one without access to banking, healthcare or government accounts.
• Start in approve-each-step mode. Where the tool offers it, have the agent ask before every action until you trust how it handles the task.
• Keep it to low-risk jobs. ASD’s advice is low-risk, non-sensitive tasks, so leave payments and private records to people.
• Stay with it while it works. Google says your active supervision is the most important protection.
• Check the result yourself. Look at the booking, form or file the agent changed, rather than trusting its summary that the job is done.
• Limit what a work agent can reach. Give it only the accounts and data the task needs, never administrator access to everything.