What's new in the Foundation Models framework

Watch on Apple Developer

TL;DR

Foundation Models expands across models and modalities

The release opens the Foundation Models framework and adds a utilities package for emerging LLM building blocks. The core theme is broader model choice, deeper OS integrations, and new primitives for agentic app experiences.

The on-device model is rebuilt with stronger logic and tool calling. Developers should use the context-size and token-counting APIs introduced in iOS 26.4 to adapt prompts, instructions, and transcripts to the current hardware. Guardrails were refined in iOS 26.4 and continue to improve in iOS 27.

The on-device model also gains image understanding through prompt attachments. Attachments can be created from UIImage, NSImage, CGImage, Core Image types, CoreVideo pixel buffers, and file URLs. Images can use arbitrary size and aspect ratio, but larger images consume more tokens and increase latency.

Inspect context size and token count

Use model-specific context and token-counting APIs before assembling prompts, instructions, or transcripts.

let model = SystemLanguageModel()
print(model.contextSize) // 8192

let count = try await model.tokenCount(
    for: "What are the Japanese characters for origami?"
)
print(count)

Attach an image to a prompt

Image attachments extend the existing prompt-builder style for multimodal prompting.

let response = try await session.respond {
    "What animal is this?"
    Attachment(UIImage(...))
}

Private Cloud Compute and model abstraction

PrivateCloudComputeLanguageModel gives apps access to Apple server models through Foundation Models. It has a 32,000-token context window and supports reasoning levels, where deeper reasoning can improve answer quality at the cost of more compute.

Using PCC does not require app-managed account setup, authentication, or API keys. The session notes that prompts are not stored, privacy claims are independently verifiable, and PCC enables Foundation Models on watchOS 27. PCC requires an entitlement, and Apple points developers to the dedicated PCC session for details.

The new LanguageModel protocol opens the model abstraction layer. SystemLanguageModel and PrivateCloudComputeLanguageModel already conform, and Apple is open sourcing CoreAILanguageModel and MLXLanguageModel for local models on Apple Neural Engine and Mac GPU. Anthropic and Google are publishing Swift packages for their models; when using third-party server models, use secure authentication such as OAuth and store tokens in Keychain, not in the app binary.

• LanguageModelSession can now be backed by local or server models through LanguageModel.
• ContextOptions(reasoningLevel:) controls how much a reasoning-capable model can think before responding.
• Sessions and responses expose usage data, including total input/output tokens, cached input tokens, and reasoning tokens.
• Third-party providers may involve separate authentication and per-token billing.

Use reasoning and inspect token usage

Reasoning and usage APIs help tune quality, latency, and cost across Apple and third-party models.

let response = try await session.respond(
    to: "Recommend a craft that doesn't require scissors.",
    contextOptions: ContextOptions(reasoningLevel: .light)
)

print(response.usage.input.totalTokenCount)
print(response.usage.input.cachedTokenCount)
print(response.usage.output.totalTokenCount)
print(response.usage.output.reasoningTokenCount)

Built-in system tools for vision and local RAG

Foundation Models adds system-provided tools backed by other OS frameworks. BarcodeReaderTool reads information from barcodes, and OCRTool extracts structured text from images. These tools complement native image understanding when a model needs specialized visual processing.

A Spotlight-powered search tool enables fully local Retrieval-Augmented Generation. The session describes RAG as providing the model with up-to-date personal or domain knowledge through a Spotlight index and processed queries, without requiring a remote retrieval service.

• Use BarcodeReaderTool for barcode information in images.
• Use OCRTool when structured text extraction improves visual reasoning.
• Use the Spotlight search tool to build local RAG over indexed content.
• For deeper implementation details, Apple points to image-understanding and Core Spotlight LLM search sessions.

Dynamic Profiles for agentic sessions

Dynamic Profiles are a declarative primitive for changing the active context of a LanguageModelSession. Instead of manually juggling multiple sessions, a DynamicProfile describes the currently active Profile: its instructions, tools, model, and configuration.

The craft app example switches between a craft-analysis mode and a brainstorming mode. Analysis uses instructions and tools to inspect journal images and save detected craft details. Brainstorming can switch to Private Cloud Compute with deep reasoning while preserving conversation history.

A DynamicProfile resolves to one active Profile at any given time. Conditionals choose the active branch, and the framework handles the transition. Apple recommends considering privacy boundaries, model capabilities, and cost when profiles switch models or reasoning levels.

Define a simple profile

A DynamicProfile can provide the current instructions and tools when creating a session.

struct CraftProfile: LanguageModelSession.DynamicProfile {
    var body: some DynamicProfile {
        Profile {
            Instructions {
                """
                You are an expert crafting assistant.
                Record craft project image analyses
                using the recordImageAnalysis tool.
                """
            }
            RecordImageAnalysisTool()
        }
    }
}

let session = LanguageModelSession(profile: CraftProfile())

Switch profiles and configure model behavior

Profile branches can change instructions, tools, model, and reasoning level while maintaining a single session history.

struct CraftProfile: LanguageModelSession.DynamicProfile {
    let states: CraftProjectStates

    var body: some DynamicProfile {
        switch states.mode {
        case .craftAnalysis:
            Profile {
                Instructions { /* ... */ }
                RecordImageAnalysisTool()
                SwitchModeTool(states: states)
            }
        case .brainstorm:
            Profile {
                Instructions { /* ... */ }
                BrainstormRecordTool()
            }
            .model(states.privateCloudCompute)
            .reasoningLevel(.deep)
        }
    }
}

Evaluation, command-line, Python, and open source support

The new Evaluations framework measures the quality of intelligence features. It is intended to quantify accuracy as prompts change and help developers understand the statistical impact of prompt and model changes before shipping.

On macOS 27, the fm command line tool exposes Apple Foundation Models from the terminal. It supports on-device and PCC usage, including interactive experimentation with fm chat and shell-script workflows for summarization, extraction, generation, and image-based file naming.

The Foundation Models Python SDK exposes the same on-device model as the Swift framework for data scientists and researchers. The open source Foundation Models framework utilities package adds building blocks such as transcript-management profile modifiers, a skill API for procedural knowledge loading, and a Chat Completions-compatible language model. The core framework is also being open sourced so Foundation Models can run wherever Swift runs, including Linux servers.

• Use Evaluations to compare prompt and behavior changes statistically.
• Use fm for terminal workflows and quick model experiments on macOS 27.
• Use the Python SDK when working in Python-centric research or data workflows.
• Use utilities for emerging LLM patterns that may evolve between OS releases.

Use the Foundation Models Python SDK

The Python SDK provides availability checking and response generation against the on-device model.

import apple_fm_sdk as fm

model = fm.SystemLanguageModel()

is_available, reason = model.is_available()
if is_available:
    session = fm.LanguageModelSession(model=model)
    response = await session.respond(prompt="Hello!")
    print(response)

Resources

• Expanding generation with tool calling
• Analyzing images with multimodal prompting
• Composing dynamic sessions with instructions and profiles
• Adding server-side intelligence with Private Cloud Compute

Unofficial, not associated with Apple. By