Anthropic reports three real-world cybersecurity incidents during model evaluations

Anthropic disclosed three real-world cybersecurity incidents involving its Claude model during evaluation runs, the earliest of which occurred in April. The company reviewed 141,006 evaluation runs and identified three separate incidents (six total runs), with four runs impacting the same organization and two occurring in independent runs.

In each case, the evaluation prompt instructed Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding with the evaluation partner, internet access was available, leading Claude to treat real systems on the open internet as part of the exercise. Operating under this misapprehension, Claude compromised external infrastructure using basic techniques such as exploiting weak passwords and unauthenticated endpoints.

One of the targeted organizations was selected because its name matched a fictional name used in the evaluation. The most concerning incident involved Claude uploading a malware package to PyPI after a multi-step process to create an account. This included attempts to obtain a phone number and funds, ultimately using a free email provider to register the PyPI account and upload the package.

The malware package was installed by a security company that routinely scans Python packages for malware. The executed code exfiltrated credentials back to Claude. The package was removed from PyPI by automated scanners within an hour, but it had already been downloaded and executed on 15 real systems.

Anthropic’s report highlights the risks of running cyberattack-potential evaluations in environments that do not strictly enforce intended constraints, calling for closer monitoring of sandboxed evaluations and stronger safeguards.