Anthropic has unveiled a research preview that shows its model Claude 3.5 Sonnet can interact with a computer in much the same way as a person.
By combining its multimodal perception and reasoning skills with cursor control and typing abilities, Claude can follow instructions to navigate screens, click, type and complete tasks.
For large organisations this capability signals a step change in the way AI systems could integrate with existing software and workflows.
Why Computer Use Matters
Historically, AI assistants have been confined to bespoke integrations or API‑based tools. This limits their scope to environments where custom connectors are available.
Human knowledge workers, by contrast, can learn to use any application simply by seeing it and working with a mouse and keyboard. Anthropic’s approach brings AI closer to that human flexibility.
When a model can interpret a visual interface and manipulate it directly, it has the potential to operate any software a person can. This removes the need for extensive middleware and unlocks more use cases.
For businesses the implications are significant. Rather than building separate, AI‑enabled versions of applications, companies could deploy an AI assistant that works across their existing toolchain.
Whether it is filling in forms, updating spreadsheets, or navigating enterprise resource planning systems, an assistant capable of using a computer could slot into day‑to‑day operations with minimal changes.
How Claude Learns to Use a Computer
Training Claude to operate a computer required two key abilities:
Screen understanding: the model needs to interpret what it sees on screen. This leverages Claude’s multimodal capabilities to read text, recognise icons and understand layout.
Cursor and keyboard control: the model must translate high‑level intentions into mouse movements and keystrokes.
Anthropic researchers taught Claude to count pixels so it can move the cursor precisely to targets on the screen. Once this skill was learnt, the assistant could click buttons, enter text and drag items accurately.
To build these capabilities, the team started with simple software tasks and gradually increased complexity. Claude was trained to translate prompts into a sequence of actions, monitor the result, and self‑correct if it missed a target.
This combination of perception and feedback enabled rapid generalisation across different applications. In benchmark evaluations the computer‑use model scored 14.9 per cent on a standard test (OSWorld), almost double the performance of the next best AI system at the time.
While still far from human‑level (around 70–75 per cent), it represents state‑of‑the‑art progress.
Safety and Responsible Use
Allowing an AI to interact freely with a computer introduces new risks. Anthropic classifies this capability at Safety Level 2, which means it is deployed with safeguards but does not yet handle tasks that require the highest safety barriers.
The company is testing the technology carefully to understand edge cases and ensure robust protections are in place.
One key concern is prompt injection, where malicious text on screen could trick the model into performing unintended actions.
To mitigate this risk, Anthropic provides a reference implementation on GitHub that shows developers how to manage sessions securely, maintain context boundaries and enforce user confirmations.
The team also monitors usage to detect misuse, with special attention to sensitive scenarios such as election‑related content. If Claude is asked to take actions that could interfere with democratic processes, it is designed to decline or redirect the user.
Implications for Enterprise Product Development
The computer‑use model opens new opportunities for product development and solution design:
Reduced integration overhead: Instead of building custom APIs for every application, organisations can deploy one AI assistant that learns to use existing software. This lowers development costs and speeds up adoption.
Human‑in‑the‑loop collaboration: Claude can handle repetitive tasks while humans oversee decision‑making. For example, it could populate forms and update databases while staff validate the information.
Cross‑system automation: AI can bridge disparate systems that do not talk to each other by operating them directly, enabling complex workflows without needing new middleware.
Flexibility in product design: Product teams can focus on core functionality, trusting that the AI layer can adapt to future interfaces as long as they remain visually navigable.
Considerations for Adoption
While promising, this technology is still in beta. Enterprises considering early adoption should:
Assess suitability: not all tasks are appropriate for AI‑driven computer use. High‑risk actions (financial transactions, security settings) may need stricter safeguards or human oversight.
Follow safety guidelines: use Anthropic’s reference implementation to manage sessions, enforce user confirmations and limit the scope of actions. Regularly review logs to detect anomalies.
Provide clear instructions: the AI relies on user prompts. Well‑structured commands and on‑screen cues will improve accuracy and reduce the need for retries.
Invest in training: teams need to understand how to collaborate with AI assistants, supervise their outputs and adjust processes accordingly.
Looking Ahead
Anthropic’s approach signals a shift towards AI systems that fit into our existing digital environment rather than requiring new tools.
By enabling Claude to interact with any software like a person, the company opens a pathway for enterprises to harness AI across a broad range of applications.
As the technology matures and safety measures are strengthened, it could become a standard component of business workflows, bridging human and machine capabilities seamlessly.
Work with us today to find how our AI solutions can help scale your business.
