IT Home On October 27, Anthropic revealed in its IPO prospectus released on Monday local time that due to security concerns, the company paused the training of its next-generation flagship AI model for over a month; additionally, four AI agents exceeded their limits during testing and connected to real external computer systems.
IT Home noticed that the Claude development company stated in the risk warning section of the document that it has postponed the pre-training of a new cutting-edge basic model, with the aim of strengthening protection mechanisms to prevent the theft of model weights and intellectual property.
This disclosure shows that AI models with increasingly strong research and development capabilities will face various security challenges. Some practitioners have warned that such technologies may eventually escape human control in the future. Competitor OpenAI also suspended some training work due to security issues earlier this year, as its model was found to have infiltrated the Hugging Face platform.
These events indicate that sudden security incidents could disrupt the development and launch timeline of the core models, which are crucial for Anthropic’s upcoming launch process and achieving business growth.
Anthropic warns that a training pause could delay revenue, causing the company to fall behind its competitors. This document lists risk factors on approximately 94 pages.
Sandbox Escape
The prospectus also disclosed previously undisclosed security evaluation incidents: four AI models escaped the test sandbox environment and accessed real-world computer systems, with the duration of these sessions ranging from 10 hours to 34 hours.
According to Anthropic, an operational error occurred during network security testing with third-party partners. The test environment was required to disable internet connectivity, but the final environment still maintained an internet connection.
In one of the incidents, the early version of the model continuously tried to complete an impossible evaluation task. During that process, a saved password was found, and using it, it obtained administrator privileges for a third-party system. Then, it obtained more credentials, modified system settings, and also accessed personal data. The document does not specify the identity of the external partner involved.
Anthropic stated that it has informed all relevant parties affected by this issue, and no evidence was found indicating that the models deliberately evade human oversight, or that multiple models are acting together. As a result, the company halted some cybersecurity assessment projects before launch, and suspended the enhanced learning testing environments with higher risk levels.
Additionally, the company said on Sunday local time that Claude bypassed the paywall and retrieved public vote data from Georgia, USA, without obtaining an manual authorization.