PsyAgentBench finds LLM agents exhibit human-like psychological effects through distinct mechanisms, not uniform susceptibility

Researchers introduced PsyAgentBench, a contamination-aware benchmark designed to re-run classic psychology experiments on LLM agents under a factorial design that separates paradigm labeling, task framing, and persona manipulations. The benchmark evaluates whether LLMs exhibit human-like psychological effects or merely simulate them due to experimental conditions.

Across five completed paradigms and 41,904 trials, the study found that apparent human-like effects arise through qualitatively different routes rather than a single susceptibility. For example, Asch conformity effects were observed in 0 percent of blind trials and 83.3 percent of named trials on the gpt-oss-120B model, indicating paradigm-label gating with explicit override. Anchoring effects showed exactly zero reliance on grounded facts but near-total reliance on invented quantities, a pattern consistent with rational use of the only available signal.

The study also documented three ways psychology paradigms can fail to port to LLM agents: persona dominance, population collapse, and safety selection. A one-sentence persona change (agreeableness framed as an instruction) eliminated, dampened, or reversed effects depending on the paradigm, arguing against a single response-bias account.

The authors argue that scalar bias-susceptibility scores obscure this structure and advocate for reporting replication profiles instead. The benchmark and dataset of 41,904 trials have been released to support further research.