Explore enhancements to visionOS object tracking

Watch on Apple Developer

TL;DR

What changed for object tracking

visionOS object tracking still starts from a USDZ model trained into a reference object, then uses ARKit anchors to give apps the physical object's position and orientation. visionOS 27 expands that model to support moving and handheld objects, including higher-frame-rate tracking for objects that move through space.

The session positions object tracking as the best fit when accuracy and precision matter, especially for measuring physical spaces or distances. If a photorealistic model of the object is unavailable, a developer can instead train and mount a marker object to the item being tracked.

• Enable high-frame-rate tracking per reference object with ReferenceObject.Configuration.
• Use the new extended training mode in Create ML for better accuracy and robustness, especially with handheld objects.
• Use metric-space poses when absolute physical coordinates matter, rather than visually corrected poses optimized for rendering.
• The same trained reference object can be used by both visionOS and iOS apps.

Enable high-frame-rate tracking for a reference object

High-frame-rate tracking is configured when loading a reference object; it is not a training-time setting and can be applied based on the app's needs.

var configuration = ReferenceObject.Configuration()
configuration.highFrameRateTrackingEnabled = true

let refObjURL = Bundle.main.url(
    forResource: "flashlight",
    withExtension: ".referenceobject"
)!
let refObject = try await ReferenceObject(
    from: refObjURL,
    configuration: configuration
)

Training and pose accuracy

Create ML gains an extended training mode for object tracking. It is recommended alongside high-frame-rate tracking when robustness is important, but it takes significantly longer than standard training.

ARKit's coordinate-space correction API lets apps choose between poses optimized for visual alignment and poses in metric space. Rendered poses keep virtual content aligned with the camera imagery, while uncorrected poses are intended for measurement and physical-coordinate calculations.

• Use standard or extended training mode in the Create ML Object Tracking template.
• Use the command-line workflow when training should run outside the Create ML app, including on a remote machine.
• Use .rendered for mixed-immersion rendering alignment.
• Use .none for metric-space measurements, distances between tracked objects, or physical placement calculations.

Train a reference object with extended mode from the command line

Extended training can be selected from the command-line object tracker workflow as well as from the Create ML app.

xrun createml objecttracker \
  --source flashlight.usdz \
  --output flashlight.referenceobject \
  --training-mode extended \
  --all-angles

Request rendered or metric-space object poses

Use corrected poses for visual alignment and uncorrected metric poses for measurement workflows.

let renderingPose = myObjectAnchor.coordinateSpace(correction: .rendered)
let metricPose = myObjectAnchor.coordinateSpace(correction: .none)

Object tracking on iOS 27

iOS 27 adds support for reference objects in ARKit APIs. Reference-object training is not platform-specific, so the same .referenceobject files can be shared between visionOS and iOS apps.

On iOS, apps load ARReferenceObject files, configure ARWorldTrackingConfiguration, and receive ARObjectAnchor updates through ARSessionDelegate. The API distinguishes mostly stationary objects from moving objects that should be tracked at high frame rate.

• Assign mostly stationary reference objects to detectionObjects.
• Assign moving reference objects to trackingObjects.
• Handle didAdd, didUpdate, and didRemove to create, update, hide, or remove associated RealityKit content.

Configure object tracking on iOS

Use detectionObjects for low-frame-rate stationary detection and trackingObjects for high-frame-rate tracking of moving objects.

let stationaryObject = try ARReferenceObject(
    archiveURL: Bundle.main.url(
        forResource: "stationary",
        withExtension: "referenceobject"
    )!
)
let movingObject = try ARReferenceObject(
    archiveURL: Bundle.main.url(
        forResource: "moving",
        withExtension: "referenceobject"
    )!
)

let configuration = ARWorldTrackingConfiguration()
configuration.detectionObjects = [stationaryObject]
configuration.trackingObjects = [movingObject]

arView.session.delegate = self
arView.session.run(configuration)

Custom spatial accessories

A spatial accessory is an electronic device that Vision Pro tracks in real time. It must include a visible LED constellation, an IMU, and Bluetooth communication; it can also include inputs such as buttons or a touchpad and outputs such as haptics.

Spatial accessories are aimed at faster, lower-latency, more robust tracking than vision-only object tracking, including temporary occlusion handling and lower-light scenarios. They are a better fit for fast-moving interactive objects or physical controls that need buttons and haptic feedback.

• Distribute LEDs to create a distinct pattern from expected viewing angles.
• Rigidly fix the LEDs and IMU to the board for accurate tracking.
• For handheld accessories, place LEDs where users are unlikely to cover them and consider battery size and ergonomics.
• For larger or distant accessories, account for LED count, size, and spacing so Vision Pro can track them accurately.
• Consult the Spatial Accessories chapter of the Accessory Design Guidelines for Apple Devices for detailed requirements and reference designs.

Validate and prepare a spatial accessory

The validation workflow starts by pairing the accessory over Bluetooth with Vision Pro, then using the ARKit accessory tracking debug view on a device in Developer Mode. The debug view helps inspect the LEDs as seen by the headset's IR camera, validate IMU frequency, latency, axis values, scale, alignment, and motion response, and debug timing using the headset's IR illuminators as a sync reference.

To make an accessory trackable by apps, create an annotated USDZ that includes a photorealistic model plus the IMU and LED positions, then generate a reference accessory file. Accessory manufacturers bundle that file in an app and declare it as an exported UTType so it is registered system-wide. App developers using a third-party accessory can bundle the file and declare it as an imported type for app-local independence.

• Use the ARKit accessory tracking debug view from Settings while Vision Pro is in Developer Mode.
• Generate a .referenceaccessory bundle from an annotated USDZ of the accessory design.
• Export the UTType when distributing the accessory definition as the manufacturer.
• Import the UTType when bundling a third-party accessory definition only for your app.
• DFRobot and MIKROE reference hardware and development kits are called out as plug-and-play options for testing and app development.

Connect accessories in an app

Apps discover generic spatial accessories through GCSpatialAccessory from Game Controller. ARKit resolves the .referenceaccessory bundle when creating an Accessory, then tracks it with AccessoryTrackingProvider.

visionOS 27 also adds an updateAccessories method so an app can switch or hot-swap tracked accessories without restarting the ARKit session.

• Use GCSpatialAccessory.spatialAccessories for discovery.
• Create an ARKit Accessory from the discovered device.
• Run an AccessoryTrackingProvider in an ARKit session.
• Call updateAccessories to change tracked accessories while the session continues running.

Discover and track a spatial accessory

Accessory(device:) resolves the reference accessory bundle automatically, and updateAccessories supports changing accessories without interrupting the running ARKit session.

import ARKit
import GameController

if let device = GCSpatialAccessory.spatialAccessories.first {
    let accessory = try await Accessory(device: device)
    let provider = AccessoryTrackingProvider(accessories: [accessory])
    try await arkitSession.run([provider])
}

try await provider.updateAccessories([newAccessory])

Resources

• Working with generic spatial accessories
• Preparing spatial accessories for tracking in your visionOS app
• Spatial accessory design guidelines for Apple devices (section 20)
• Exploring object tracking with ARKit

Related Sessions

• Explore spatial accessory input on visionOS
• Explore object tracking for visionOS

Unofficial, not associated with Apple. By