微软 Copilot 引入确定性信任控制层防御提示注入

As organizations adopt generative AI into daily workflows, they face a new class of security challenges that do not fit traditional models of access control, identity, or data protection. One of the most important is indirect prompt injection: malicious instructions embedded in otherwise legitimate content such as emails, documents, or messages that an AI system may process as part of a task.

When that content is passed to the model alongside trusted prompts, the system may treat embedded instructions as commands rather than data. This can influence model behavior and contribute to outcomes such as unintended data disclosure, inaccurate information in responses, and other harms.

These attacks can occur entirely within existing permissions. The system is authorized to access the content, and traditional controls such as authentication, access governance, and data loss prevention may still function as designed while leaving inference-time risk unaddressed.

The Structural Gap in AI Systems

Traditional computing systems use layered structural controls such as hardware isolation, privileged instruction sets, and access control models to reduce exposure to specific classes of attack vectors.

In many generative AI architectures, instructions and data are represented as tokens within a single representational input stream. Unlike traditional computing systems, neural networks do not independently provide equivalent structural separation between instructions and data. This architecture difference fundamentally changes the security model: malicious instructions embedded in untrusted grounding content will be processed alongside the original instructions.

Industry guidance reflects the severity of this issue: the OWASP Top 10 for LLM Applications 2026 ranks prompt injection as the number one application-level security risk for LLM applications. Existing mitigations such as classifier-based filtering, input sanitization, and hardened system prompts can reduce exposure to indirect prompt injection. Many prompt injection-related security vulnerabilities are difficult to mitigate with point fixes and may regress under variations of attack patterns, presenting an ongoing security risk. These techniques reduce risk but no single mitigation prevents every indirect prompt injection attempt. As part of our defense in depth strategy, our new approach complements these mitigations by adding a deterministic enforcement layer at inference time.

A New Direction: Deterministic Trust-Aware Controls

Recent research has explored architectural approaches that apply pre-defined constraints to content based on its origin. Instead of relying solely on probabilistic detection, these approaches introduce deterministic controls that preserve clear trust boundaries. Read the research.

This approach is designed to introduce a deterministic enforcement layer in M365 Copilot between retrieved content (tool outputs) and the primary large language model. This layer establishes a trust boundary designed to specifically handle content from an untrusted source before it enters the model context.

How Trust-Aware Processing Works

Trust-Aware Content Handling

When M365 Copilot retrieves grounding data, the system uses deterministic signals to categorize content as trusted or untrusted.

For example, external content such as emails from outside the organization can be classified by policy as untrusted when crossing the organizational trust boundary. This classification provides a practical, scalable signal that can help reduce prompt injection risk.

Privilege Separation via Quarantine & Constrained Processing

Untrusted content is routed through a quarantined, capability-constrained processing path rather than being provided directly to the primary model. There, it is processed in the context of the user’s query into a transformed representation. The transformation seeks to retain useful information while dropping malicious content, reducing the surface area by limiting the information that can be presented to the primary model.

This reduces the surface area available to adversarial instructions and limits the information that can be carried into downstream reasoning.

Provenance-Aware Processing

The system can add explicit provenance markers to indicate when content originates from an untrusted source. Maintaining this origin information allows downstream processing to account for content provenance and helps responses reflect the trust characteristics of their inputs.

Safer Synthesis

The primary model generates responses using:

• Trusted content (passed through unchanged)
• Transformed, structured representations of untrusted content

This reduces exposure of the primary model to untrusted inputs, reducing the likelihood that adversarial instructions can influence outcomes while preserving task-relevant information to maintain utility.

Example: Email-Borne Prompt Injection

Consider a routine business email that contains hidden or embedded instructions. When a user asks their AI assistant to summarize their inbox, the system retrieves the email and passes the email content into the model alongside system prompts. Embedded malicious instructions may then influence model output.

In these cases there is no violation of access policy or mishandling of data; the system is functioning as designed, but it can fail to distinguish data to be summarized from instructions to be executed at inference time.

How Deterministic Controls Help Reduce Indirect Prompt Injection Risk

This architecture applies consistent controls at defined points in the orchestration pipeline. While traditional approaches attempt to identify malicious behavior within a shared input space, this approach constrains how information flows through that space.

The enforcement layer provides:

• Deterministic signals: Untrusted content is intercepted at a fixed point in the orchestration pipeline using deterministic signals.
• Structured hand-off: Untrusted content is transformed, removing malicious content while retaining benign content based on the user's prompt.

By reducing both the fidelity and control surface of untrusted inputs, this approach lowers the probability that indirect prompt injection attacks can succeed, even when the system is authorized to access the underlying data.

Looking Ahead

Microsoft 365 Copilot is introducing a new enforcement layer. Initial investments focus on high-risk content, including external communications. We are exploring expanded coverage for additional content types and sources, along with deeper integration with enterprise signals and policy systems to support more granular trust models.

To configure this protection in your tenant, enable the applicable Outlook sender setting used to identify untrusted content and review the configuration guidance for CVE-2026-55145. Availability and behavior may vary by release status and tenant environment.

The rise of indirect prompt injection attacks highlights a broader shift in AI security. This work brings foundational security principles – explicit trust boundaries, constrained processing, and deterministic enforcement – into the AI inference layer.

As generative AI continues to evolve, these architectural advances can help enterprise AI systems become more secure, reliable, and trustworthy by design.