Controlled experiment shows humans shift strategy when playing against LLMs in competitive games

A controlled laboratory experiment examined how human subjects behave when competing strategically against large language models compared to other humans. The study used a p-beauty contest game—a standard economics framework where payoffs depend on the median choice across all players—and offered monetary incentives to participants.

The researchers found a notable behavioral divergence: when playing against LLMs, human subjects selected significantly lower numbers than when competing against other humans. This shift was primarily attributable to increased choices of the Nash equilibrium value of zero. The effect concentrated among participants with higher demonstrated strategic reasoning ability, suggesting that more sophisticated players consciously adjusted their approach when facing AI opponents.

When asked to explain their strategies against LLMs, subjects cited two key beliefs: that the AI would reason rationally through the game, and that it would cooperate. This dual expectation—rational calculation paired with collaborative intent—appears to have driven their decision-making, even though these assumptions warrant scrutiny in the context of how language models actually operate.

The findings open questions about mechanism design in environments where humans and LLMs make simultaneous choices. If humans systematically adjust their behavior based on imperfect mental models of AI reasoning and intent, system designers may face hidden incentive misalignments or cascading strategic errors. The research is presented as a preprint and contributes foundational empirical data on a largely unexplored interaction pattern.