AI safety advocates debate scope of risks and policy response amid underinvestment concerns

Arvind Narayanan and Sayash Kapoor argue that the AI safety debate is polarized between two narratives: that existential risk from AI is imminent, or that warnings are insincere attempts at regulatory capture. They propose a third view—that some warnings are sincere but misguided—and call for a more nuanced approach to safety that focuses on well-grounded risks rather than speculative ones.

The authors highlight systemic underinvestment in preparedness for catastrophic risks, pointing to pandemic response as a case study. Despite repeated warnings and quantitative estimates—such as a 2019 WHO/World Bank report projecting a respiratory pathogen killing 50 to 80 million people—the majority of pandemic preparedness recommendations have gone unimplemented, even after COVID-19. They argue that voters and policymakers prioritize relief over prevention, leaving societies vulnerable to foreseeable disasters.

Narayanan and Kapoor extend this critique to cybersecurity, noting that markets and institutions systematically underprice tail risks. They cite Lloyd’s of London’s 2022 assessment that losses from severe state-backed cyberattacks could exceed what the insurance market can absorb, emphasizing that this problem predates recent AI capabilities. They argue that catastrophic cyber risks—such as cascading internet failures—are not speculative but plausible, and deserve serious study by the security community, which has historically neglected them.

The authors also address diffuse, gradual risks, such as the erosion of traditional journalism due to AI-powered chatbots reducing clickthrough rates. They describe this as an example of “accumulative risk,” where societal resilience erodes incrementally and unnoticed. Such risks, they argue, are as deserving of policy attention as acute catastrophes, but are harder to mobilize action around due to their pervasive and slow-moving nature.

Narayanan and Kapoor critique the longtermist and expected-utility frameworks often used to justify existential risk framing in AI safety. While they endorse consequentialist reasoning for risks like biorisk, they argue that extending these methods to longtermism and AI existential risk leads to misplaced urgency. They also reject the tendency to reframe systemic problems—such as AI’s role in amplifying biorisk—as “AI safety” problems solvable through technical alignment, instead viewing AI as an amplifier of existing risks.

The authors question whether more urgent public warnings about AI risks actually lead to better policymaking, citing the Coxon incident as an example of how media distortions can elevate existential risk concerns over more grounded issues. They argue that the AI community’s intense focus on AI’s transformative potential is not shared by the broader public, where AI ranks as a minor concern in polls, suggesting a disconnect between advocacy and public priorities.

Ultimately, Narayanan and Kapoor advocate for a “big-tent” approach to AI safety that prioritizes evidence-based, systemic risk governance over speculative existential threats. They call for policies that address real underinvestment in resilience, whether in pandemic preparedness, cybersecurity, or the preservation of societal institutions, while cautioning against rhetoric that distracts from actionable priorities.