AI leaders at The Curve discuss limits on model intelligence amid recursive self-improvement concerns

At The Curve, an annual conference in Berkeley, executives from leading AI labs, nonprofit leaders, government officials, and journalists discussed whether to impose hard limits on how intelligent AI systems can become. Speakers suggested that restrictions could include de-facto bans on systems reaching superhuman intelligence, though the feasibility and definition of such limits remain under debate.

The push for limits comes amid growing concern about recursive self-improvement—systems that can research and train their successors—amid recent blog posts from OpenAI and Anthropic outlining progress toward this capability. Some speakers argued that without limits, labs risk losing control over increasingly autonomous systems.

Proposed approaches to cap intelligence included restricting the use of frontier models for AI research, limiting compute availability, capping the number of model copies a system can run, and preventing deployment of models beyond a certain capability threshold. However, enforcement mechanisms for these restrictions do not yet exist, and individual labs or countries cannot impose them unilaterally.

The US government currently opposes such restrictions, according to the report, while lab leaders warn of potential catastrophe as soon as next year. This divergence underscores ongoing tension between industry self-regulation and government policy, particularly under the current administration, which has alternately pursued a licensing regime for frontier models and encouraged faster development.

Anthropic CEO Dario Amodei has previously called for a 'speed limit' on recursive self-improvement, and Anthropic’s responsible scaling policy has been adopted in some form by leading rivals. Other proposals, such as embedded evaluators and antitrust waivers for safety collaboration, were discussed but deemed insufficient by some speakers.

The conference also highlighted broader disagreements over the proximity of existential risk, with lab leaders expressing greater urgency than government officials. The debate over intelligence limits reflects a growing recognition that current governance mechanisms may not address the risks posed by increasingly capable and autonomous systems.