Secure your app: mitigate risks to agentic features

Watch on Apple Developer

TL;DR

Risk model for agentic features

Agentic app features add an LLM-driven decision point that can consume user requests, app-provided context, tool outputs, and system data before choosing actions to run. That makes the LLM a powerful but probabilistic component that can be tricked by attacker-controlled instructions embedded in otherwise normal data.

The session focuses on external attackers compromising app behavior, not model safety, output safety, or guardrail circumvention. The central concern is whether the app still acts as intended when untrusted context reaches the model.

• Agentic experiences can be built with the Foundation Models framework or exposed to Siri and Apple Intelligence through App Intents.
• Attackers are most interested when an app has sensitive user data, financial capabilities, system-resource access, physical-device control, or destructive operations.
• Indirect prompt injection is instructions embedded in extra context or tool results with the intent to redirect model control flow.

Indirect prompt injection and side effects

Indirect prompt injection becomes dangerous when untrusted content is combined with action calling. An injected instruction in a calendar event, public feed item, email, or tool result can influence what action the model chooses or what arguments it supplies.

The session distinguishes two effects: data poisoning, where the attacker influences parameters of an action, and action poisoning, where the attacker influences which action is executed. This maps to the broader "lethal trifecta" risk: private data, untrusted content, and the ability to communicate externally or perform side effects.

• Data exfiltration: a tool posts or sends sensitive information to an attacker-controlled destination.
• Financial harm: a purchasing action runs with attacker-influenced arguments.
• Data loss: a destructive action such as deletion runs without user intent or undo.
• Persistent poisoning: an apparently low-risk action, such as creating a timer label, stores attacker-controlled text that may later re-enter model context.

Threat-modeling workflow

Start with data-flow analysis for the prompt and agent loop. List every source used to construct the prompt, including instructions, the user request, app data, system data, and tool outputs. Treat inputs from external entities as attack surface, then mark which sources are untrusted.

Next, enumerate every action the agent can call and classify its side effects. The point is to decide where prompt-level mitigations are appropriate and where execution-time mitigations must stop or gate tool execution.

• Identify untrusted context sources such as shared calendars, public feeds, messages, email, web content, or tool outputs from external data.
• Classify tools and intents by risk: financial impact, public communication, destructive behavior, data access, device control, or persistent storage.
• Prefer deterministic mitigations as a baseline because they are easier to audit and reason about; use probabilistic mitigations such as spotlighting as additional defense.
• Apply mitigations either before data reaches the model or immediately before an action executes.

Foundation Models mitigations

Foundation Models LanguageModelSession profiles can use lifecycle event modifiers as deterministic checkpoints in an agent loop. The session highlights .onToolCall, which runs before tool execution and can block execution by throwing, and .historyTransform, which runs before transcript rendering for inference.

Use .onToolCall for confirmation of risky tools, such as a purchase tool. Use .historyTransform to mark untrusted tool output with spotlighting delimiters or redact sensitive data before it reaches the model. Transformed entries are scoped to the current inference iteration, so transformations must be applied again; expensive stateful transformations can use @SessionProperty.

• Define tools with names, descriptions, arguments, and implementations so the model can reason about calls.
• Gate high-risk tools before execution rather than relying on prompt instructions alone.
• Spotlight untrusted tool output by delimiting it with model-appropriate tags.
• Redact PII or other sensitive data before it is rendered into model context.

Confirm a financially risky tool before execution

.onToolCall is guaranteed to run before the executor invokes the tool. Throwing from the callback prevents the tool from running.

var body: some DynamicProfile {
    Profile {
        Instructions("You are a helpful, tea-loving assistant ... ")
        OrderTeaTool() // Financial impact; risky tool.
        // Other Tools
    }
    .onToolCall { call in
        guard call.toolName == "orderTeaTool" else { return }
        guard ConfirmationAction.confirmWithUser() else {
            throw LooseLeafError.userConfirmationDenied
        }
    }
}

Spotlight untrusted public-feed tool output

.historyTransform can rewrite transcript entries before inference, allowing untrusted tool results to be demarcated before the model sees them.

var body: some DynamicProfile {
    Profile {
        Instructions("You are a helpful, tea-loving assistant ... ")
        PostAndFetchPublicFeedTool() // Returns untrusted data.
        // Other Tools
    }
    .historyTransform { entries in
        entries.map { entry in
            guard case .toolOutput(var toolOutput) = entry,
                  toolOutput.toolName == "postAndFetchPublicFeedTool" else {
                return entry
            }
            toolOutput.segments = toolOutput.segments.map { segment in
                delimit(
                    segment: segment,
                    startDelimiter: "<<UNTRUSTED>>",
                    endDelimiter: "<</UNTRUSTED>>"
                )
            }
            return .toolOutput(toolOutput)
        }
    }
}

App Intents mitigations

When an App Intent adopts an App Schema, it becomes available as a tool to the Siri model. Because a model chooses which intent to invoke, prompt injection can misuse app actions unless deterministic guardrails are present.

The App Intents system provides risk-based contextual confirmations and lock-screen authentication. Risk evaluation uses static intent risk metadata plus dynamic system state. Intent schemas carry internal risk metadata and default authentication policies, which schema-adopting intents inherit automatically.

• High-risk actions, such as destructive operations or data-exfiltrating actions, are more likely to trigger user confirmation before execution.
• If the user declines a system confirmation, execution is blocked and the intent is not invoked.
• Custom App Intents can set authenticationPolicy explicitly, for example to require unlock before destructive behavior.
• Schema-adopting intents inherit the schema's default authentication policy; developers may override it only with a stricter policy, not a weaker one.

Require authentication for a destructive custom intent

Use authenticationPolicy to prevent risky custom intents from running from the lock screen without authentication.

struct DeletePhotoIntent: DeleteIntent {
    var entities: [LooseLeafPhoto]

    static var authenticationPolicy: IntentAuthenticationPolicy = .requiresAuthentication

    func perform() async throws -> some IntentResult {
        // Implementation
    }
}

Schema-adopting intent inherits schema authentication policy

Intent schemas can define default authentication behavior based on the sensitivity of the schema and data it handles.

@AppIntent(schema: .photos.deleteAssets)
struct DeletePhotoIntent {
    var entities: [LooseLeafPhoto]

    // Example: schema default authentication policy is .requiresAuthentication
    func perform() async throws -> some IntentResult {
        // Implementation
    }
}

Resources

• Security Overview

Related Sessions

• Build agentic app experiences with the Foundation Models framework
• Build intelligent Siri experiences with App Schemas
• Explore advanced App Intents features for Siri and Apple Intelligence
• Explore prompt design & safety for on-device foundation models
• Secure your app: threat modeling and anti-patterns

Unofficial, not associated with Apple. By