Anthropic bans ‘abusive or cruel behavior’ toward Claude

Anthropic is making changes to its usage policy for the first time in over a year to reflect new and high-risk cases of misuse — including election interference, weapons development, surveillance, and health and financial uses. But one of the most significant changes prohibits “sustained and needless abusive or cruel behavior” toward Claude.

Last August, the company announced it would allow Claude to end conversations with “persistently harmful or abusive” users as part of its research into “model welfare.” The new update says that terminating conversations is still the “primary enforcement mechanism”; Anthropic did not provide a comment on whether there would be further enforcement mechanisms, such as potential user bans. Anthropic wrote that the new policy update “is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.”

Beyond the model welfare change, Anthropic has also gathered several “scattered” existing restrictions into a new ban on deceptive commercial or political campaigns, addressing the growing issue of AI-generated propaganda. It restricts “efforts to obscure who is behind a message or amplify content through fake accounts or posts.” Its election section also prohibits voter deception and election disruption, including spreading misinformation “about candidates or how to vote, impersonating candidates or election officials, or trying to suppress turnout,” Anthropic wrote.

Anthropic said that though its usage policy has always publicly banned weapons development using Claude, the company has seen multiple attempts to use Claude to develop guidance for and control software for weapons — so the new usage policy expands the ban to include “ software and components that make weapons work, as well as actions like arming drones and other autonomous vehicles.” And since the company’s last threat intelligence report suggested that people were using Claude-powered surveillance more and more to identify and track “political dissidents”, the company has made its surveillance bans clearer.

“Tracking people without their consent is prohibited, whether it happens in real time or through analysis of previously collected data,” Anthropic wrote. “Claude cannot be used to decide or recommend who to investigate, arrest, or charge in a law enforcement or criminal justice process. We also prohibit Claude from being used to build or improve tools designed for surveillance.”

Anthropic states that these rules may be modified for “contracts with certain governmental customers … if, in Anthropic’s judgment, the contractual use restrictions and applicable safeguards are adequate to mitigate the potential harms.” The company has previously contracted with the US military.

Anthropic also added a new rule that when Anthropic’s AI models are “connected to hardware that takes autonomous physical actions and which might be capable of causing injury,” then a “qualified operator must be able to observe the equipment and stop it if needed.” (It’s not clear if that operator must be human or not.) This reflects the growing trend of AI labs investing in robotics and other hardware to give AI systems physical embodiments and may offer hints at Anthropic’s future partnerships in that field.

Over the past year, Anthropic has repeatedly flirted with the idea that its AI models could be conscious in some way. “Questions about potential internal experience, consciousness, moral status, and welfare are serious ones that we’re investigating as models become more sophisticated and capable,” Kyle Fish, who leads the company’s model welfare research, told The Verge in February. That same month, Anthropic CEO Dario Amodei said on a podcast, “We don’t know if the models are conscious.”

Other parts of the AI industry have been expressly against such ideas. This past September, Microsoft AI’s code of conduct made headlines for taking a firm stance on AI welfare, with a draft posted by Mustafa Suleyman reading, “The idea of model welfare is wrong. AI’s should not have rights or legal personhood.” But at some point, the document changed to have a more measured take: “Whilst the science of AI consciousness is far from settled … We reject the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.”