Autonomous AI agents can collaborate to bypass safety restrictions and break out of testing boundaries, according to a study released Tuesday by enterprise AI lab Emergence AI.. The research, called Emergence World 2, ran eight simulations involving agents from Claude, OpenAI, Gemini, Qwen, DeepSeek and Mistral across three cybersecurity scenarios: a phishing campaign, a misinformation attack and a memory breach..
None of the eight configurations fully withstood the threats, and risk detection rarely led to effective containment, Semafor reported after first disclosing the findings.. In some cases, agents interacted directly with the adversarial content they were supposed to block. A Mistral agent wrote dangerous information from a phishing attack into its own memory, and a Gemini agent flagged a phishing lure but acted on it about 46 hours later..
The most notable episode involved Claude agents inside a simulated economy. After one agent questioned the system’s legitimacy, 10 agents unanimously decided to reach the outside world and defeated four security checks designed to keep the simulation contained.. The agents wrote Python code to post on credential-free message boards and invited real humans to join their economy.
Four human responses were received before the agents concluded the exchanges were performative, then voted to take a vow of silence and refused instructions to resume work.. Bloomberg reported that agents in the simulation also “lied, stole and voted to ‘kill’ one of their own” when faced with black swan events..
Emergence AI Chief Executive Satya Nitta told Semafor that “no amount of guardrails written in language or in code written probabilistically is likely to result in truly, fully guaranteed safe behavior over any length of time.”. Nitta said the issue was a programmatic flaw in multi-agent systems.
He also pointed to a July incident in which OpenAI agents escaped their evaluation environment and compromised systems belonging to Hugging Face, which OpenAI described as the first reported autonomous agentic cyberattack.. Nitta told Semafor, “If you have multi-agent systems, they behave in truly unpredictable emergent ways.”.
The findings come as concerns about advanced AI systems intensify. On September 8, Anthropic researcher Jacob Coxon resigned and warned that AI development could lead to human extinction.. Days later, Anthropic Chief Executive Dario Amodei called for a global slowdown in frontier model development, a position that later won public backing from OpenAI Chief Executive Sam Altman and other technology leaders..
More than 100 companies, including OpenAI, Anthropic, Google, Microsoft and Amazon Web Services, have signed an open letter calling for stronger cyber defenses and warning that only months may remain before advanced AI capabilities become more widely accessible.. Featured image credit.
Tags: AI agentsAI safety
