Best practices for integrating visual intelligence in your app

Watch on Apple Developer

TL;DR

Integration model

Visual Intelligence integration has two main paths: provide results to Visual Intelligence through Image Search, and receive information that Visual Intelligence writes into shared system stores. Image Search uses App Intents plus the Visual Intelligence framework so the system can pass captured visual context to your app and display your app's results alongside other providers.

The sample app searches album artwork and surfaces both matching albums and related upcoming concerts. The same entity, query, and open-intent approach works across iOS, iPadOS, and macOS, with platform-specific input differences to account for.

• Use AppEntity for content that can appear in Visual Intelligence results.
• Use IntentValueQuery with SemanticContentDescriptor input to return visual search results.
• Use OpenIntent to navigate directly to the selected content.
• Use system stores such as EventKit, Contacts, and HealthKit when Visual Intelligence creates data that your app already reads.

Define result content as AppEntity

Visual Intelligence displays your app's results using AppEntity values from the App Intents framework. The entity's DisplayRepresentation is especially important because the result UI has limited space: roughly a title, subtitle, and thumbnail.

Choose identifying text that helps users recognize the result quickly. If using image URLs, serve thumbnail-sized images when returning multiple results; full-resolution assets are unnecessary for small cards and can slow loading. If your integration typically returns one result, remember the image may occupy the full width of the results sheet.

• Include a stable identifier and properties needed to display and open the content.
• Provide typeDisplayRepresentation and defaultQuery as with other AppEntity adoption.
• Keep title/subtitle concise; for the sample album entity, the key fields are album name and artist name.

Album AppEntity for Visual Intelligence results

Defines album content as an AppEntity and controls how each result appears in the Visual Intelligence sheet.

import AppIntents

struct AlbumEntity: AppEntity {
    var id: String
    @Property var name: String
    @Property var artistName: String
    var coverArtData: Data

    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(artistName)",
            image: .init(data: coverArtData)
        )
    }

    static let defaultQuery = AlbumEntityQuery()
    static var typeDisplayRepresentation: TypeDisplayRepresentation { "Album" }
}

Implement fast, ranked image search

An IntentValueQuery is the lightweight query entry point Visual Intelligence uses to ask your app for results. For visual search, the system passes a SemanticContentDescriptor containing information about the captured image, including a pixel buffer when available.

The sample performs on-device image similarity with Vision feature prints. It precomputes feature prints for catalog entries, generates a feature print for the captured image at query time, filters by a maximum distance threshold, sorts by similarity, and returns a limited set of top results.

The main best practices are independent of whether matching runs locally or on a server: keep queries fast, rank the most relevant results first, limit result count, and return an empty array if there are no good matches.

• Guard for missing pixelBuffer and return no results rather than failing the experience.
• Precompute expensive catalog representations such as feature prints outside the query path.
• Apply a distance or relevance threshold so weak matches do not clutter results.
• Consider other Vision APIs for app-specific search, such as text extraction, barcode scanning, face detection, and image classification.

IntentValueQuery using SemanticContentDescriptor

Receives captured visual context and returns app-specific Visual Intelligence results.

import AppIntents
import VisualIntelligence

struct SearchHandler: IntentValueQuery {
    @Dependency var catalog: AlbumCatalog

    func values(for input: SemanticContentDescriptor) async throws -> [VisualSearchResult] {
        guard let pixelBuffer = input.pixelBuffer else { return [] }
        let albums = try await catalog.search(matching: pixelBuffer)
        return albums.map { VisualSearchResult.album($0) }
    }
}

Vision feature-print matching

Converts the captured pixel buffer, compares it to precomputed catalog feature prints, ranks by similarity, and limits results.

import Vision

private func generateFeaturePrint(for image: CGImage) async throws -> FeaturePrintObservation {
    let request = GenerateImageFeaturePrintRequest()
    return try await request.perform(on: image)
}

func search(matching pixelBuffer: CVReadOnlyPixelBuffer,
            limit: Int = 10,
            maxDistance: Double = 1.0) async throws -> [AlbumEntity] {
    var cgImage: CGImage?
    _ = pixelBuffer.withUnsafeBuffer {
        VTCreateCGImageFromCVPixelBuffer($0, options: nil, imageOut: &cgImage)
    }
    guard let cgImage else { return [] }

    let queryPrint = try await generateFeaturePrint(for: cgImage)
    return try entries.compactMap { entry -> (AlbumEntity, Double)? in
        let distance = try queryPrint.distance(to: entry.featurePrint)
        guard distance <= maxDistance else { return nil }
        return (entry.album, distance)
    }
    .sorted { $0.1 < $1.1 }
    .prefix(limit)
    .map { $0.0 }
}

Open results and continue search in your app

When the user taps a Visual Intelligence result, the system invokes an OpenIntent for the selected entity. Reuse an existing OpenIntent if your app already defines one for App Intents or Siri; a Visual Intelligence-specific duplicate is not needed.

Keep the open intent lightweight because it runs while the app is foregrounding. Navigate to the right screen, then defer heavier loading until after the destination view appears.

For cases where the first result is not enough, adopt the semanticContentSearch schema. This gives users a "More results" path into your full in-app search experience, with the same SemanticContentDescriptor available so you can prepopulate filters, categories, and results from the captured context.

• OpenIntent should land users directly on the selected content, not a generic home screen.
• semanticContentSearch should preserve context from Visual Intelligence instead of starting a blank search.
• Your in-app search can show richer UI than the Visual Intelligence result sheet, including filters and deeper catalog navigation.

OpenIntent for a selected album

Navigates directly to the selected album when a Visual Intelligence result is tapped.

import AppIntents

struct OpenAlbumIntent: OpenIntent {
    static let title: LocalizedStringResource = "Open Album"

    @Parameter(title: "Album") var target: AlbumEntity
    @Dependency var appState: AppState

    func perform() async throws -> some IntentResult {
        await appState.openAlbum(id: target.id)
        return .result()
    }
}

Continue captured-context search in app

Uses the semantic content search schema to open a full in-app search seeded from the captured image.

@AppIntent(schema: .visualIntelligence.semanticContentSearch)
struct SemanticContentSearchIntent: AppIntent {
    static let title: LocalizedStringResource = "Search in app"
    static let openAppWhenRun: Bool = true

    var semanticContent: SemanticContentDescriptor
    @Dependency var catalog: AlbumCatalog
    @Dependency var concertFinder: ConcertFinder
    @Dependency var appState: AppState

    func perform() async throws -> some IntentResult {
        guard let pixelBuffer = semanticContent.pixelBuffer else { return .result() }
        let albums = try await catalog.search(matching: pixelBuffer)
        let artists = albums.map { $0.artistName }
        let concerts = await concertFinder.findNearby(byArtists: artists)
        await appState.openSearch(albums: albums, concerts: concerts)
        return .result()
    }
}

Multiple result types and cross-platform input

An app can have only one IntentValueQuery that accepts SemanticContentDescriptor. To return more than one kind of result from that query, define a @UnionValue type with a case for each result entity. The sample combines visually matched albums with nearby concerts derived from the matched artists.

Visual Intelligence is available on iOS, iPadOS, and macOS. The same AppEntity, IntentValueQuery, and OpenIntent code can carry over with minimal changes, but the captured input may differ. iPhone usage often starts from the camera and physical objects, while iPad and Mac primarily use screenshots and digital media. On Mac, captured pixel buffers can be much larger, so resizing may be appropriate before processing.

• Use @UnionValue when one visual query should return heterogeneous entity types.
• Provide an OpenIntent for each entity type you return.
• Think beyond pixel matches: use visual context to derive related app content such as events, products, or recommendations.
• Test both camera-like captures and screenshot captures, especially for large macOS pixel buffers.

UnionValue for albums and concerts

Returns multiple visual search result types from the single SemanticContentDescriptor query.

@UnionValue
enum VisualSearchResult {
    case album(AlbumEntity)
    case concert(ConcertEntity)
}

struct SearchHandler: IntentValueQuery {
    @Dependency var catalog: AlbumCatalog
    @Dependency var concertFinder: ConcertFinder

    func values(for input: SemanticContentDescriptor) async throws -> [VisualSearchResult] {
        guard let pixelBuffer = input.pixelBuffer else { return [] }
        let albums = try await catalog.search(matching: pixelBuffer)
        let artists = albums.map { $0.artistName }
        let concerts = await concertFinder.findNearby(byArtists: artists)
        return albums.map { .album($0) } + concerts.map { .concert($0) }
    }
}

System store integrations

Visual Intelligence can also create data in shared system stores that apps may already read. Events created from detected event information are available through EventKit, contact information through Contacts, and medical-device readings through HealthKit.

The sample music app reads calendar events to show upcoming concerts. It requests calendar access, fetches events in the near future, filters for event titles that match known artists, and observes EventKit store changes so newly added Visual Intelligence events appear automatically.

• Use EKEventStore for events saved by Visual Intelligence.
• Use CNContactStore for contacts added from captured information such as business cards.
• Use HKHealthStore for supported medical-device readings, such as blood pressure monitor, glucose meter, or weight scale displays.
• Observe store-change notifications where available so app UI updates when Visual Intelligence adds new data.

Fetch upcoming calendar events with EventKit

Reads calendar events that may have been created by Visual Intelligence and filters them into app-specific upcoming concerts.

import EventKit

@Observable
class UpcomingConcertManager {
    private let eventStore = EKEventStore()
    var upcomingConcerts: [EKEvent] = []
    var authorizationStatus: EKAuthorizationStatus = .notDetermined

    func requestAccessAndFetch() async throws {
        let granted = try await eventStore.requestFullAccessToEvents()
        guard granted else {
            authorizationStatus = .denied
            return
        }
        authorizationStatus = .fullAccess
        await fetchUpcomingConcerts()
    }

    func fetchUpcomingConcerts() async {
        let predicate = eventStore.predicateForEvents(
            withStart: .now,
            end: .now.addingTimeInterval(90 * 24 * 60 * 60),
            calendars: nil
        )
        let events = eventStore.events(matching: predicate)
        upcomingConcerts = events.filter { event in
            AlbumCatalog.shared.entries.contains { entry in
                event.title?.localizedCaseInsensitiveContains(entry.album.artistName) == true
            }
        }
    }
}

Observe EventKit changes

Refreshes app data when the EventKit store changes, including events newly created through Visual Intelligence.

for await _ in NotificationCenter.default.notifications(named: .EKEventStoreChanged) {
    await fetchUpcomingConcerts()
}

Resources

• Integrating your app with visual intelligence
• Visual Intelligence

Unofficial, not associated with Apple. By