Create robust evaluations for agentic apps

Watch on Apple Developer

TL;DR

Why synthetic evaluation data matters

The Evaluations framework, new in Xcode 27, helps assess intelligence-powered Swift app features, track quality over time, and support production confidence across macOS, iOS, watchOS, and visionOS.

The BookTracker example starts with 13 hand-written Book samples for an auto-tagging feature. That is enough to begin development, but too narrow to represent the range of books, genres, review styles, vague summaries, and edge cases users may provide.

The session frames synthetic data as an iterative workflow: start with seed samples, generate more, validate the results, analyze representativeness, and repeat until the dataset covers meaningful real-world variation.

• Small datasets can produce misleadingly high evaluation scores.
• Synthetic data generation is defined in code, can run from command-line workflows, and supports structured data through the Generable macro.
• The useful question is not just "how many samples?" but whether the dataset covers the meaningful ways the feature will be used.

Generate synthetic samples with makeSamples and SampleGenerator

makeSamples takes a generation prompt, an existing dataset, and a targetCount. The target count is the final dataset size including seed samples, so a target of 100 from 13 seeds generates 87 new samples.

For more control, SampleGenerator lets the app define a sessionProvider that returns a LanguageModelSession. This controls the model and system instructions used for generation. The session is reused across batches, but if the context window is exhausted, the provider may be called again, so instructions should be self-contained.

samplingStrategy controls which seed examples are shown in context. Random sampling is the default and works for unordered datasets; sliding-window sampling is useful when the seed dataset has meaningful order.

• Default generation uses the on-device model.
• A custom session can use PrivateCloudComputeLanguageModel when larger context is useful.
• Batch size is handled by the framework.
• Generation returns an async stream of new samples.

Expand a dataset with makeSamples

Uses seed ModelSample values and a prompt to stream generated samples until the final dataset reaches the requested target count.

let prompt = Prompt("""
Generate diverse range of book reviews and corresponding tags.
Cover a wide range of genres, time periods, cultures, and reader personas.
Do not repeat books already in the dataset.
""")

let dataset = Book.sampleBooks.map { book in
    ModelSample(prompt: book.review, expected: BookTags(tags: book.tags))
}

let targetCount = 100
var expandedDataset = dataset

for try await sample in dataset.makeSamples(prompt, targetCount: targetCount) {
    expandedDataset.append(sample)
    print("Generated \(expandedDataset.count) samples so far.")
}

Customize generation with SampleGenerator

Creates a generator with an explicit model and task instructions instead of relying on the default configuration.

let generator = SampleGenerator<ModelSample<BookTags>>(
    prompt,
    samples: dataset,
    targetCount: targetCount,
    sessionProvider: {
        LanguageModelSession(
            model: PrivateCloudComputeLanguageModel(),
            instructions: """
            You are a synthetic data generator for a book-tracking app's evaluation suite.
            Rules:
            - Review must be at least 100 characters long.
            - Review should cover a mix of genre, mood/tone, and themes.
            - Reviews should vary in length.
            - Create between 3 and 8 tags.
            - Tags must be lowercase.
            """
        )
    }
)

Validate generated data before trusting it

Prompt instructions are not guarantees. The validator closure lets the evaluation pipeline accept or reject each generated sample using deterministic checks that can be applied to one sample at a time.

In BookTracker, the validator checks review length, tag count, and lowercase tags. Broader qualities like diversity across the entire dataset require separate review because the per-sample validator does not see other generated samples.

Valid generated samples are collected in samples; rejected outputs are collected in invalidSamples. Both are updated during the run and can be inspected while generation is running or after it completes.

• Use validators for systematic rules, not holistic judgments requiring cross-sample context.
• Save or reuse only the samples that satisfy the dataset contract.
• Inspect invalid samples to improve prompts, instructions, or validators.

Validate synthetic BookTracker samples

Rejects generated samples that do not meet the app's structural requirements.

validator: { sample in
    guard let book = sample.expected else { return false }

    guard sample.promptDescription.count >= 100 else { return false }
    guard (3...8).contains(book.tags.count) else { return false }
    guard book.tags.allSatisfy({ $0 == $0.lowercased() }) else { return false }

    return true
}

Access valid and invalid results

Reads accepted and rejected generated samples from the generator after or during a run.

for try await sample in generator.run() {
    expandedDataset.append(sample)
}

let allSamples = await generator.samples
let invalidSamples = await generator.invalidSamples
print("Generated \(allSamples.count) new samples. Total: \(expandedDataset.count)")

Use evaluation comparisons to find weak spots

The Xcode 27 Evaluations Report can compare evaluation runs. In the BookTracker example, the 13-sample run shows high tag relevance and usefulness scores, while the 100-sample run drops after adding broader synthetic coverage.

A score drop is useful signal, not automatically a failure of one component. It may indicate problems in the prompt, instructions, intelligence feature implementation, evaluation criteria, or the representativeness of the dataset itself.

• Compare small and expanded dataset runs to detect overconfidence from narrow coverage.
• Use score changes to decide whether to refine prompts, change feature logic, adjust evaluators, or generate more edge cases.
• Treat dataset expansion as part of the same develop-and-evaluate loop as model and prompt iteration.

Evaluate agentic workflows with trajectory expectations

For agentic features, final output quality is not enough. A model can produce a plausible answer while using the wrong tool, skipping a required lookup, using bad arguments, or calling a tool the user explicitly asked it not to use.

The BookTracker library assistant uses tools such as searchBooks, getBookDetails, and findSimilarBooks. Tool evaluations check the path through these tools: correct tool names, argument values or intent, ordering, and disallowed calls.

TrajectoryExpectation describes expected tool calls in a language model session transcript. ToolCallEvaluator combines a LanguageModelSession with the app's tools, captures the structured transcript, and reports results alongside other evaluations in Xcode.

• Use unordered expectations when only the presence of a tool call matters.
• Use ordered expectations when one tool must precede another, such as searching before requesting details for a bookId.
• Use argument matchers such as .exact, .naturalLanguage, .contains, .oneOf, .pattern, and .range depending on how strict the expected argument should be.
• Use disallowed expectations for tools that must not appear in the transcript.

Define a tool argument type

Uses a Generable struct so the model can populate optional tool arguments based on the user request.

@Generable
struct SearchBooksArguments {
    @Guide(description: "A freeform search term to match against titles, reviews, or tags")
    var query: String?

    @Guide(description: "Filter results to books with this specific tag")
    var tag: String?

    @Guide(description: "Filter results by mood")
    var mood: String?

    @Guide(description: "Filter results by genre")
    var genre: String?

    @Guide(description: "Maximum number of results to return. Defaults to 5.")
    var limit: Int?
}

Check ordered, argument-aware tool calls

Requires the assistant to search for gothic books before requesting details for a returned book.

TrajectoryExpectation(
    ordered: [
        ToolExpectation(
            "searchBooks",
            arguments: [
                .exact(argumentName: "tag", value: .string("gothic"))
            ]
        ),
        ToolExpectation(
            "getBookDetails",
            arguments: [
                .keyOnly(argumentName: "bookId")
            ]
        )
    ]
)

Disallow an unwanted tool call

Verifies that a search happens while ensuring the model does not call findSimilarBooks.

TrajectoryExpectation(
    unordered: [
        ToolExpectation(
            "searchBooks",
            arguments: [
                .naturalLanguage(
                    argumentName: "genre",
                    criteria: "Should refer to science fiction"
                )
            ]
        )
    ],
    disallowed: [
        ToolExpectation("findSimilarBooks")
    ]
)

Synthesize datasets for tool-call evaluations

The same synthetic-data workflow can expand tool-evaluation datasets because ModelSample and TrajectoryExpectation are generable. The generation prompt and session instructions should explicitly describe available tools, their purposes, order constraints, and supported argument matchers.

Validation remains important for generated tool samples. The session validates that each sample has an expectation, includes at least one expected tool, and references only real tools from the app.

Running output evaluations and tool-call evaluations in one suite gives broader confidence: one evaluation checks what the model produces, while the other checks how it gets there.

• Tell the generator what tools exist; it does not infer your app's tool list automatically.
• Include order requirements, such as requiring searchBooks before getBookDetails or findSimilarBooks.
• Validate generated expectations before using them as test data.

Validate generated tool-evaluation samples

Rejects generated trajectory samples that omit expectations, contain no tool calls, or reference tools the app does not define.

validator: { sample in
    guard sample.output.expectations != nil else { return false }
    let expectations = sample.output.expectations!

    let totalExpectations = expectations.ordered.count + expectations.unordered.count
    guard totalExpectations > 0 else { return false }

    let validTools: Set<String> = ["searchBooks", "getBookDetails", "findSimilarBooks"]
    let allExpectations = expectations.ordered + expectations.unordered + expectations.disallowed

    for expectation in allExpectations {
        guard validTools.contains(expectation.name) else { return false }
    }

    return true
}

Resources

• Book Tracker: Using Evaluations to evaluate an intelligent feature
• Generating synthetic datasets
• Evaluating tool-calling behavior
• Scoring with model-as-judge evaluators

Unofficial, not associated with Apple. By