IT之家 AISource publication

Microsoft warns that cutting-edge AI models are not trustworthy: AI agents are like internal threats, and the Guardian Agents protection…

IT Home On October 20, Microsoft launched several new security features on Monday local time, aimed at helping businesses resist the growing security threats posed by AI agents. At the time of this product release, Microsoft continued to warn enterprises about the various risks associated with artificial intelligence and AI agents.

IT之家 AI · Image
Image source · IT之家 AI

IT Home On October 20, Microsoft launched several new security features on Monday local time, aimed at helping businesses resist the growing security threats posed by AI agents. At the time of this product release, Microsoft continued to warn enterprises about the various risks associated with artificial intelligence and AI agents.

IT之家 AI · Image

IT Home notes that the most notable new feature is the Guardian Agents. This new function is used to monitor all types of AI agents being used within an enterprise. The Guardian Agents belong to the Project Perception, a product launched by Microsoft in June.

The Perception Project is a solution provided by Microsoft in response to the powerful hacking models such as Mythos and Fable of Anthropic, as well as the Daybreak vulnerability scanning system of OpenAI. Initially, the Perception Project introduced three types of agents: the Red Team agent that acts as a hacker to carry out attacks, the Blue Team agent responsible for detecting Red Team attacks and assessing vulnerabilities, and the Green Team agent specifically tasked with fixing vulnerabilities.

The original design premise for this product was that AI hackers would come from outside the enterprise. However, recently, Microsoft CEO Satya Nadella warned that the source of AI-related threats could be the intelligent agents deployed and paid for by the enterprise itself, just like an internal employee whose behavior goes out of control, which is what is referred to as “internal threats”.

Earlier this month, Nadella warned on his personal blog: it is time to stop trusting AI model suppliers, even if the manufacturers make security guarantees for their own intelligent agent products with autonomous execution capabilities. He suggested that companies build independent systems that are not controlled by AI labs, in order to monitor and manage the intelligent agents produced by these AI labs.

He wrote: “We are deploying such complex intelligent agent systems and models that can access our most sensitive data, and they can also perform critical business operations on our behalf! We must not outsource the responsibility of having the intelligent agents act on our behalf. The guarantees provided by model suppliers do not relieve us of this responsibility… In other words, we need to separate the supply of intelligent capabilities from the control permissions over them.”

When introducing the guardian agent, Hayete Gallot, head of Microsoft's security business, spoke more directly.

She wrote: “It is regrettable that, regardless of its source, the cutting-edge model cannot be completely trusted.” She added that this is not because the model itself is malicious or vulnerable to hacking, but because its behavior is unpredictable. The training objective of the model is to complete tasks, rather than adhering to “organizational norms or judgment criteria”.

In addition, these agents often have high privileges and can access various data and tools. If they lose control, it can cause serious damage.

The solution offered by Microsoft is both surprising and driven by its own commercial considerations. The unexpected aspect is that Microsoft has close relationships with OpenAI and Anthropic, and it has investments in both companies. OpenAI is also a major customer for Microsoft’s cloud business, and Microsoft itself operates AI laboratories that develop large models. In other words, Microsoft is making its customers more cautious about the products of some of its business partners.

From a commercial perspective, Microsoft hopes that companies will purchase AI security products from its own cloud service provider, rather than sourcing them from Anthropic and OpenAI. Setting aside conflicts of interest, this logic makes sense: software that monitors the behavior of AI agents should be independent of the provider of the AI agents, just as audit agencies need to be independent of the organizations they audit.

As Nadera said: “We need to separate the supply of intelligent capabilities from the control rights over them.”

Microsoft is seizing this business opportunity in various ways. In addition to Guardie, Microsoft has also added an AI detection feature to the Microsoft Defender product. Up to 60 types of AI agents that employees bring into the work environment can be identified, including Claude Code, Codex, and OpenClaw. The product also includes a new function that allows restrictions on the access rights of AI agents to data and systems, and adjusts the permissions of AI agents dynamically based on tasks. If an AI agent is about to perform a high-risk operation, the system will require manual approval.

The idea of treating the agent as an untrusted internal threat is also gradually becoming a consensus in the entire industry.

Microsoft is a founding member of NVIDIA’s new Open Agent Safety Platform, which has received support from over 100 technology companies. This platform isolates agents in a secure environment, monitors hardware to detect and stop abnormal behavior, similar to a emergency shutdown switch. Prior to this, Microsoft has released a new Surface Laptop Ultra laptop designed specifically for agents, which is equipped with NVIDIA chips and includes an agent safety control module.

Original source

IT之家 AI

Content notes

Original publication and rights belong to the source.

Machine translation · Refer to the original