Improve your prompts by hill-climbing with Evaluations

Watch on Apple Developer

TL;DR

Hill-climbing with Evaluations

The session frames prompt engineering as hill-climbing: make a targeted change, run an evaluation, analyze the results, then repeat. The approach assumes an existing evaluation pipeline with datasets, subjects, evaluators, metrics, and expectations already in place.

The running example is Book Tracker, a book cataloging app whose tagging service generates tags from reader reviews. The initial output is plausible but misses important themes and sometimes includes reader reactions or overly specific phrases that are not useful search tags.

• Use evaluation scores as a guide, not as a substitute for inspection.
• Define pass/fail expectations with Swift Testing so evaluation runs can fail when aggregate metrics fall below target.
• Use Xcode's evaluation report and assistant editor to inspect aggregate charts and per-sample model outputs against expected data.

Evaluation expectations with Swift Testing

Evaluation tests can encode acceptance criteria so hill-climbing iterations produce concrete pass/fail signal.

@Test("Book Tag Evaluations", .evaluates(evaluation))
func evaluateBookTagging() async throws {
    let result = EvaluationContext.current.result
    #expect(result.aggregateValue(.mean(of: evaluation.tagCount)) >= 0.8)
    #expect(result.aggregateValue(.mean(of: evaluation.noDuplicates)) == 1)
}

Evaluate both heuristics and qualitative quality

The BookTaggingEvaluation combines deterministic checks with model-judge scoring. Heuristics verify requirements such as 3-8 tags, at least one genre tag, and no duplicates. Qualitative score dimensions measure whether tags are relevant to the book and useful for browsing a library.

The model judge uses ScoreDimension definitions and a ModelJudgePrompt to rate generated tags. The session emphasizes that the definitions must be specific enough for the judge to apply the same standards as the developer.

• Relevance: tags should describe the book itself, such as genre, themes, tone, setting, or form.
• Usefulness: tags should work as shelf labels, broad enough to group books but specific enough to narrow search.
• Bad tags include reader reactions, meta-commentary about the review, author facts, genre contradictions, character names, made-up phrases, and hyper-specific descriptors.

Qualitative score dimensions

ScoreDimension descriptions are a high-leverage part of a model-judge evaluator because they define the rubric the judge applies.

let relevance = ScoreDimension(
    "Relevance",
    description: "Whether each tag describes the book itself - its genre, themes, tone, or setting - rather than the reader's reactions, meta-commentary, or facts about the author.",
    scale: .numeric([
        4: "Every tag describes the book itself",
        1: "Tags don't meaningfully describe the book"
    ])
)

let usefulness = ScoreDimension(
    "Usefulness",
    description: "Whether tags work as library shelf labels - broad enough that several books could share the tag, specific enough to meaningfully narrow a search.",
    scale: .numeric([
        4: "Every tag could group multiple books while still narrowing a search",
        1: "Tags would not help with browsing"
    ])
)

Detect and reduce model-judge drift

During analysis, the developer and model judge disagree on usefulness scores even when relevance scores match. This divergence is called drift: the model judge's ratings increasingly differ from expert human ratings as the dataset grows.

The session recommends aligning the judge to expert opinion before trusting it to score feature changes. Simple accuracy can be misleading when most examples are high quality or score distributions are uneven, because a judge can appear aligned by guessing common high scores. Cohen's kappa accounts for chance agreement and is used as the alignment metric.

• Build a judge-alignment evaluation from the same review/tag pairs that both the human expert and model judge rate.
• Add expert ratings to the extracted dataset, then have the model judge rate the same generated tags.
• Aggregate mean, standard deviation, and a custom Cohen's kappa score for each dimension.
• Use an alignment target of 0.6 as a meaningful agreement threshold in this example.

Custom Cohen's kappa aggregation

A custom aggregation compares expert scores with judge scores to quantify alignment instead of relying on raw average scores.

func aggregateMetrics(using aggregator: inout MetricsAggregator) {
    let expertRelevance = Self.samples.map { Double($0.expected?.expertRelevanceScore ?? 0) }

    aggregator.group("Relevance") { group in
        group.computeMean(of: relevance.metric)
        group.computeStandardDeviation(of: relevance.metric)
        group.custom(of: relevance.metric, label: "Relevance Alignment Score") { judge in
            cohensKappa(ratings1: expertRelevance, ratings2: judge) ?? 0
        }
    }
}

Judge calibration expectation

The calibration test fails until the judge prompt and scoring dimensions are refined enough to match the expert's rubric.

@Test("Judge Calibration", .evaluates(evaluation))
func evaluateJudgeCalibration() async throws {
    let result = EvaluationContext.current.result
    #expect(result.aggregateValue(.custom(label: "Relevance: Judge vs Expert")) > 0.6)
    #expect(result.aggregateValue(.custom(label: "Usefulness: Judge vs Expert")) > 0.6)
}

Use comparative evaluation like a controlled experiment

Xcode 27 can compare two evaluations side by side. The session treats this like a controlled experiment: the baseline evaluation is the control, and a second evaluation with one intentional change is the experimental group.

The first experimental judge prompt adds more app context and examples of good and bad tags. Relevance alignment improves, but usefulness alignment drops, illustrating that prompt changes can introduce tradeoffs. Subsequent iterations isolate one variable at a time by applying the new prompt to the baseline before changing only the ScoreDimension descriptions.

Few-shot worked examples are added only after prompt and rubric refinements still fall short. The examples show how the expert scores several tag sets, but the session cautions to keep the example set small to avoid overfitting the alignment score.

• Compare baseline and experimental evaluations in one test suite.
• Inspect per-sample differences in Xcode's comparison view to find failure patterns.
• When changing scoring dimensions, keep the prompt constant so the effect of the rubric change is isolated.
• Use worked examples to calibrate the judge, but do not simply train it to memorize the calibration dataset.

Experimental judge prompt structure

The improved prompt gives the judge app context, separate dimensions, positive examples, and known failure modes.

ModelJudgeEvaluator(
    judge: .default,
    dimensions: [relevance, usefulness],
    prompt: ModelJudgePrompt(
        instructions: """
        You are an experienced reader and librarian evaluating tags automatically generated for Book Tracker.
        Score the tag set on two independent dimensions: Relevance and Usefulness.
        ## What a good tag looks like
        - Genre/form, theme/subject, tone/atmosphere, setting/era
        ## Common failure modes
        - Reader reactions, meta-commentary, author facts, genre contradictions
        """,
        evaluationTarget: { output in output.tags.joined(separator: ", ") },
        reference: { input, _ in ["Book Review": input.promptDescription] }
    )
)

Hill-climb beyond prompts with tools

After aligning the judge, the session applies it to the actual Book Tracker tag generator. The next feature change is not a prompt change: the on-device model receives a BookLookupTool that can provide title and author from distinguishing review details.

BookTaggingService gains a tools parameter with an empty-array default so the existing evaluation path remains unchanged. A second evaluation passes the lookup tool and is compared against the no-tool baseline. The tool-enabled version meets expectations and performs better in the small dataset, but the session notes that 13 samples is not enough coverage and that tool calls should be evaluated directly.

• Tools, models, instructions, datasets, aggregations, and evaluators are all valid hill-climbing variables.
• Keep existing tests stable by defaulting new tool parameters to an empty array.
• For agentic behavior, evaluate whether the tool was called in the right situations, not just whether final output improved.
• The session points to tool call evaluators and the Sample Generator API for broader agentic app coverage.

BookTaggingService with tools

Adding a tools parameter lets the same service run both the baseline and tool-enabled variants.

struct BookTaggingService {
    static func generateTags(for review: String, tools: [any Tool] = []) async throws -> BookTags {
        let session = LanguageModelSession(
            model: SystemLanguageModel(guardrails: .permissiveContentTransformations),
            tools: tools,
            instructions: instructions
        )
        let response = try await session.respond(to: tagsPrompt(review: review), generating: BookTags.self)
        return response.content
    }
}

Evaluation variant with lookup tool

The experimental evaluation differs from the baseline only by passing the BookLookupTool.

struct BookTaggingWithLookupEvaluation: Evaluation {
    func subject(from sample: ModelSample<BookTags>) async throws -> ModelSubject<BookTags> {
        let result = try await BookTaggingService.generateTags(
            for: sample.promptDescription,
            tools: [BookLookupTool()]
        )
        return ModelSubject(value: result)
    }
}

Resources

• Book Tracker: Using Evaluations to evaluate an intelligent feature
• Designing effective model-as-judge evaluators
• Designing specific, measurable criteria in an evaluation suite

Unofficial, not associated with Apple. By