Extend Reality Composer Pro 3 functionality with Xcode

Watch on Apple Developer

TL;DR

Plugin architecture and team workflow

Reality Composer Pro 3 supports Xcode-built plugins so engineers can expose project-specific data and behavior directly inside the editor. Artists and designers can edit custom components, preview runtime behavior, and iterate without repeatedly building and deploying the app.

A typical project keeps the Reality Composer Pro project and Xcode project in the same git repository. The Xcode project builds both the final app and a plugin framework, while Reality Composer Pro authors scenes and exports Reality Files consumed by the app.

• Use the editor's simulation bar / Run With Xcode workflow to link the Reality Composer Pro project with the Xcode project.
• The Xcode project can have separate schemes for the app and the plugin framework, sharing custom RealityKit code between both.
• Imported assets are converted into Reality Composer Pro's internal JSON-backed data format; the editor also provides a custom merge tool intended to reduce content merge conflicts.
• When opening a project with plugins, Reality Composer Pro asks the user to trust the plugin before loading it.
• After rebuilding a plugin, restart Reality Composer Pro to load the updated dynamic library.

Register custom components and systems

The cauldron example starts with a custom Cauldron component that stores a water level. Components exposed to Reality Composer Pro 3 must conform to Codable so they can be represented in the editor and serialized to Reality Files.

A custom RealityKit System queries entities with the component and runs inside the editor through the plugin. In the example, it finds a child water mesh and changes its position based on the component's inspector-editable waterLevel property.

• Implement custom data as a RealityKit Component.
• Implement editor/runtime behavior as a RealityKit System.
• Register both from a type conforming to RealityComposerProPlugin.
• Export createRealityComposerProPlugin with @_cdecl so the plugin loader can create the plugin instance.
• Debug plugin code by setting breakpoints in Xcode and attaching to the Reality Composer Pro application.

Minimal custom component and system

Defines editor-serializable component state and a system that applies it live to the scene.

import RealityKit

public struct Cauldron: Component, Codable {
    public var waterLevel: Float
}

public struct CauldronSystem: System {
    let query = EntityComponentQuery(Cauldron.self)

    public init(scene: Scene) {}

    public func update(context: SceneUpdateContext) {
        for (entity, cauldron) in context.entities(matching: query) {
            guard let water = entity.findEntity(named: "Cauldron_Water_mesh") else { continue }
            water.setPosition(SIMD3<Float>(0, 1, 0) * cauldron.waterLevel,
                              relativeTo: entity)
        }
    }
}

Reality Composer Pro plugin registration

Registers custom RealityKit types with the editor and exposes the plugin entry point.

import RealityComposerPro

final class RCPCustomComponentsPlugin: RealityComposerProPlugin {
    public func setup(context: any RealityComposerProContext) {
        context.registerComponent(Cauldron.self)
        context.registerSystem(CauldronSystem.self)
    }
}

@_cdecl("createRealityComposerProPlugin")
public func createRealityComposerProPlugin() -> UnsafeMutableRawPointer {
    return RCPCustomComponentsPlugin().passRetained()
}

Drive Shader Graph materials from component data

The session extends the component with vortex-related properties, then updates the system so those properties drive a ShaderGraphMaterial. This lets a tech artist build the visual shader in Reality Composer Pro while engineers expose higher-level controls in Swift.

The system retrieves the water mesh's ModelComponent, casts the first material to ShaderGraphMaterial, computes derived surface parameters, sets named shader parameters, and writes the modified material back to the model.

• Use component properties such as rotationSpeed, minWaterLevel, maxWaterLevel, and vortexCoeff as editor-facing controls.
• Use ShaderGraphMaterial.setParameter(name:value:) to pass computed values into the shader graph.
• After changing the component's stored properties, Reality Composer Pro shows a change-acceptance dialog when the rebuilt plugin is loaded.

Update Shader Graph parameters from a RealityKit system

Shows the core pattern for propagating component state into a Shader Graph material each update.

guard var model = water.components[ModelComponent.self] else { continue }
guard var mat = model.materials.first as? ShaderGraphMaterial else { continue }

let surface = computeSurface(cauldron: cauldron)
try? mat.setParameter(name: "Level Radius", value: .float(surface.levelRadius))
try? mat.setParameter(name: "Lowest Point", value: .float(cauldron.waterLevel - surface.lowestPoint))
try? mat.setParameter(name: "Height Change", value: .float(surface.heightChange))
try? mat.setParameter(name: "Level Coeff", value: .float(surface.levelCoeff))
try? mat.setParameter(name: "Is Level", value: .bool(surface.isLevel))

model.materials[0] = mat
water.components.set(model)

Add custom animation actions to the sequencer

Reality Composer Pro 3's animation sequencer can use custom animation actions defined in a plugin. The cauldron example implements SetWaterLevelAction, which animates the component's waterLevel between start and end values on a sequencer timeline.

The action conforms to EntityAction and Codable. Execution is implemented by subscribing to EntityAction .updated events, computing normalized playback time, interpolating the water level, and writing the updated component back to the target entity.

• Return an animatedValueType; the example uses Transform.self to access the target entity in the animation executor.
• Register the action with context.registerAction.
• Call the action's subscription setup from plugin loading code so animation updates actually execute the action.
• In the editor, the custom action appears in the sequencer and can be dragged onto an animation track.

Custom EntityAction for water-level animation

Defines the serializable action parameters that appear in the sequencer inspector.

import RealityKit

public struct SetWaterLevelAction: EntityAction, Codable {
    public let startWaterLevel: Float
    public let endWaterLevel: Float

    public var animatedValueType: (any AnimatableData.Type)? {
        Transform.self
    }
}

Subscribe to animation updates and register the action

Executes the custom action during playback and makes it available to Reality Composer Pro.

extension SetWaterLevelAction {
    static func subscribe() {
        Task { @MainActor in
            SetWaterLevelAction.subscribe(to: .updated) { event in
                let normalizedTime = (event.playbackController.time - event.startTime) / event.duration
                let action = event.action
                let currentLevel = action.startWaterLevel +
                    Float(normalizedTime) * (action.endWaterLevel - action.startWaterLevel)

                guard let entity = event.targetEntity else { return }
                guard var cauldron = entity.components[Cauldron.self] else { return }
                cauldron.waterLevel = currentLevel
                entity.components.set(cauldron)
            }
        }
    }
}

public func setup(context: any RealityComposerProContext) {
    context.registerComponent(Cauldron.self)
    context.registerSystem(CauldronSystem.self)
    context.registerAction(SetWaterLevelAction.self)
    SetWaterLevelAction.subscribe()
}

Expose custom types to Script Graph

Plugins can also add custom Script Graph nodes so designers can use project-specific Swift types in no-code graphs. The quickest path shown is annotating a custom component with @Scriptable, which generates a schema for scripting.

Register scripting modules on the main thread using RealityKitScripting. Once registered, the editor can show nodes for the custom component, such as setting the cauldron water level in response to keyboard input in a Script Graph.

• Import RealityKitScripting and RealityKitScriptingMacros.
• Add @Scriptable to the custom component type.
• Create an RKS.Configuration with one or more Module definitions.
• Add the generated schema, such as Cauldron.SchemaProvider.schema, to the module.
• Call RKS.addConfiguration from the plugin setup path on the main actor.

Mark a component scriptable

Generates a schema that can be registered with the Script Graph system.

import RealityKit
import RealityKitScripting
import RealityKitScriptingMacros

@Scriptable
public struct Cauldron: Component, Codable {
    public var waterLevel: Float
    public var rotationSpeed: Float
    public var minWaterLevel: Float
    public var maxWaterLevel: Float
    public var vortexCoeff: Float
}

Register a scripting module

Makes the generated custom component schema available as Script Graph nodes in the editor.

Task { @MainActor in
    let config = RKS.Configuration(id: "ChaparralVillage")
        .onInitialize { _ in
            [
                Module("ChaparralVillage") {
                    Cauldron.SchemaProvider.schema
                }
            ]
        }

    try! RKS.addConfiguration(config)
}

Related Sessions

• Design no-code games with Reality Composer Pro 3
• Explore advances in RealityKit
• Iterate your spatial scenes faster with Reality Composer Pro 3
• Supercharge your spatial workflows with Reality Composer Pro 3

Unofficial, not associated with Apple. By