AI guardrails complicate offensive cybersecurity research, researchers say

OpenAI and Anthropic have created vetted access programs—OpenAI’s Trusted Access for Cyber and Anthropic’s Cyber Verification Program—to allow approved researchers to use their models with fewer cybersecurity restrictions. These programs were designed to balance access with safety, but researchers say the guardrails are impeding their work.

Security researchers who probe systems for unknown vulnerabilities and develop exploits rely on AI tools to assist in tasks such as confirming bugs and reverse engineering. When guardrails block these activities, defenders lose a critical capability, according to Chris Anley, chief scientist at NCC Group. He described AI models as dual-use tools—essential for both offensive and defensive security work—comparing them to a hammer that is both a construction tool and a potential weapon.

Some researchers, including those at CrowdStrike subsidiary CrowdFense, avoid using frontier models for sensitive tasks like finding vulnerabilities or building exploits due to concerns about data leakage or exposure in future training runs. Instead, they use open-source models run locally, which come without guardrails but require more operational overhead.

A researcher at a smartphone-component manufacturer, speaking anonymously, said their employer’s tools are barely usable for security-related tasks because guardrails are too strict, blocking workflows when they detect potential security activity.

Chris Thompson, CEO of RemoteThreat and founder of Offensive AI Con, reported that guardrails in frontier models are inconsistent and change frequently, forcing researchers to spend time negotiating with models rather than analyzing vulnerabilities. This inconsistency pushes some researchers toward Chinese open-source models like GLM, which can be run locally without restrictions.

Anthropic’s models Mythos and Fable were temporarily subject to U.S. export controls in June, which were later lifted—Fable 5 returned to general access on July 1, and Mythos 5 was reintroduced only to vetted U.S. organizations as part of a government review process.