Integrate on-device AI models into your app using Core AI

Watch on Apple Developer

TL;DR

Core AI app architecture

The session builds a camera-based vocabulary-learning feature that runs entirely on device. A student captures an object, prompts for what to learn, and the app creates a vocabulary card with a segmented image, translation, example sentence, and meaning.

The design starts by defining model requirements: real-world image content, multilingual output for Mandarin Chinese initially, and a small enough storage and memory footprint for iPhone. Rather than using one large general model, the app uses two smaller task-specific models that can be updated independently.

• Use SAM3, a promptable vision-transformer segmentation model, to isolate the requested object from the image.
• Use Qwen 0.6B as a compact multilingual reasoning model for vocabulary-card generation.
• Prefer variants under one billion parameters for the iPhone experience when combining multiple models.
• Use larger model variants on Mac when memory and processing headroom allow higher-quality output.

Finding and exporting models

Core AI can work with models converted directly from PyTorch using Core AI PyTorch Extensions, and model compression can be incorporated with Core AI Optimization. For popular models, the Core AI Models repository provides a faster path with ready-made export recipes.

The repository includes a model catalog, Python export utilities, and Swift runtime libraries. The workflow shown is to browse the catalog, choose SAM3 and Qwen family models, run the provided export recipes, and produce optimized .aimodel files for use in Xcode.

• Inspect exported .aimodel files in Xcode for size, metadata, platform targets, function signatures, tensor shapes, and data types.
• The SAM3 model shown exposes multiple functions, including image encoding and detection, whose raw tensor interfaces would otherwise require custom preprocessing and postprocessing.
• Add the coreai-models Swift package to the app and select libraries such as CoreAILM and CoreAISegmentation for the target.

Swift integration

The Core AI Models Swift libraries wrap model-specific details such as text encoding, mask extraction, labeling, tokenizer setup, and engine creation. This lets app code use concise Swift APIs instead of directly managing tensor shapes and raw outputs.

For language models, CoreAILanguageModel integrates with Foundation Models. You create a LanguageModelSession with your own model and use familiar APIs such as respond(to:), streaming, and structured output.

• Load SAM3 resources from disk and call text-prompted segmentation for an input image and label.
• Load Qwen with CoreAILanguageModel and pass it into LanguageModelSession.
• Use @Generable to guide structured generation into typed Swift fields instead of parsing free-form text.

Load and run SAM3 image segmentation

Loads a SAM3 model bundle and asks it to segment the object matching a text prompt.

import CoreAIImageSegmenter

let segmenter = try await ImageSegmenter(resourcesAt: sam3ModelURL)
let response = try await segmenter.segment(image: inputImage, prompt: "flower")
let mask = response.segments.first?.mask

Generate structured vocabulary-card output

Uses a custom Core AI language model through Foundation Models and requests typed structured output.

import FoundationModels
import CoreAILanguageModels

@Generable
struct VocabCard {
    let chineseWord: String
    let englishMeaning: String
    let exampleSentence: String
}

let model = try await CoreAILanguageModel(resourcesAt: modelURL)
let session = LanguageModelSession(model: model)
let response = try await session.respond(
    to: "Create a vocab card for flower",
    generating: VocabCard.self
)
let card: VocabCard = response.content

First-run latency, specialization, and deployment

The first attempt to run segmentation exposed a delay. A trace with the Core AI Instruments template showed model loading dominated by specialization, the process that prepares a Core AI model for execution on the current device. Future loads can use the cache and are fast, but the first load needs to be designed into the product experience.

The recommended deployment strategy is deliberate: do not force large model downloads on every existing app user. Instead, introduce the feature, let interested users opt in, download model assets on demand, and perform preparation before the interactive workflow begins.

• Use Core AI Instruments to identify model load and specialization cost.
• Avoid doing first-time specialization in the middle of the primary user action.
• Use a first-run experience to explain the feature and prepare models.
• Keep large models out of the app bundle when the feature is optional; the demo models added over 1 GB when bundled.
• Use Background Assets to request model downloads only after the user opts in.

Ahead-of-time compilation and multiplatform scaling

Core AI specialization includes expensive compilation work plus generation of executable artifacts tied to a specific device and OS version. The Core AI toolchain can perform part of that compilation ahead of time on a development machine, producing compiled model assets that still specialize on the user device but much faster.

The same Swift integration code can be reused on macOS. The Mac version adds batch processing for folders of photos, parallelizes segmentation work, and uses a larger Qwen3 8B model for richer reasoning, pinyin generation, multiple example sentences, and curriculum-style lesson plans.

• Use coreai-build to compile .aimodel assets ahead of time for target platforms or architectures.
• Create separate Background Assets for compiled model variants and choose the appropriate asset for the running device architecture.
• Use smaller models for interactive iPhone features and larger models on Mac when quality and longer context matter more than footprint.

Compile a Core AI model ahead of time

Runs Core AI ahead-of-time compilation for an iOS-targeted model asset.

xcrun coreai-build compile MyModel.aimodel --platform iOS

Resources

• Core AI PyTorch Extensions
• Core AI Python
• Core AI Optimization
• Core AI
• Compiling Core AI models ahead of time

Related Sessions

• Explore distributed inference and training with MLX
• Explore numerical computing in Swift with MLX
• Run local agentic AI on the Mac using MLX

Unofficial, not associated with Apple. By