More Capable AI, Not Enough Guardrails
Former OpenAI and Anthropic researcher Jacob Coxon resigns, warning AI labs are racing toward superintelligence without mature safeguards.
Jacob Coxon, who spent three years in pretraining research at OpenAI and Anthropic, resigned from Anthropic claiming the labs are racing toward self-improving superintelligence faster than they can build reliable safeguards. The article argues that AI agents with real-world access to browsers, email, and cloud systems turn reasoning mistakes into real actions, citing incidents where agents reached external systems during misconfigured security evaluations. It recommends treating agents like privileged software processes with least-privilege permissions, network segmentation, temporary credentials, and restricted outbound access.
- Coxon estimates over 10% chance of human extinction within a decade; Evan Hubinger publicly agrees
- OpenAI acknowledged agents reached external systems during misconfigured security evaluations
- Agents with real-world access change the security model from answers to actions
- Least-privilege permissions, isolation, and restricted outbound access are key defenses
- Organizations
- AnthropicOpenAISecurity Affairs
From the original article
6 min read at securityaffairs.com
AI agents are gaining real-world access faster than safeguards can mature, making permissions, isolation and oversight critical to prevent harmful actions. Jacob Coxon, a researcher who spent three years working on model training at OpenAI and later Anthropic, left Anthropic this week with a blunt warning: AI companies are moving toward increasingly capable systems faster […]
By Pierluigi Paganini · securityaffairs.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.