Explore advances in RealityKit

Watch on Apple Developer

TL;DR

Lighting and shadows

RealityKit's lighting updates focus on making virtual content blend more believably with virtual and physical environments. Lightmaps can represent complex static lighting effects such as indirect lighting, ambient occlusion, and beauty textures; the recommended workflow is to generate them with the Reality Composer Pro 3 light baker.

Dynamic lights gain soft shadows by configuring the shadow light size and quality. A larger light size produces a wider penumbra, but soft shadows require medium or high quality; low quality produces hard shadows regardless of light size.

Projective textures let spotlights project texture patterns, such as stars, caustics, or window patterns. Physical space lighting allows supported virtual lights to affect system environments or the real world through RealityKit scene understanding; the session states this is currently supported for spotlights and point lights.

• Use lightmaps for static indirect lighting, ambient occlusion, and precomputed beauty lighting.
• Use soft shadows on dynamic lights when the virtual light has a nonzero area and the cost is acceptable.
• Use SpotLightComponent.ProjectiveTexture to project patterns from a spotlight.
• Add SpotLightComponent.SurroundingsLight to enable physical space lighting for a spotlight.

Enable soft shadows on a spotlight

Set a nonzero lightSize and use medium or high shadow quality to get soft shadow penumbra from a dynamic spotlight.

guard var shadow = hearthSpotlight.components[SpotLightComponent.Shadow.self] else {
    // handle error
}

shadow.lightSize = 0.7      // meters
shadow.quality = .medium    // .medium or .high enables soft shadows
hearthSpotlight.components.set(shadow)

Project a texture and enable physical space lighting

Attach a projective texture to a spotlight, then add SurroundingsLight so the light can interact with the physical environment.

let spotLightEntity = Entity()
spotLightEntity.components.set(SpotLightComponent(
    color: .white,
    intensity: intensity,
    innerAngleInDegrees: innerAngle,
    outerAngleInDegrees: outerAngle,
    attenuationRadius: attenuationRadius
))

let projectiveTexture: TextureResource = generateStarsAndNebulaeTexture()
spotLightEntity.components.set(SpotLightComponent.ProjectiveTexture(
    texture: projectiveTexture
))

spotLightEntity.components.set(SpotLightComponent.SurroundingsLight())

Navigation mesh pathfinding

RealityKit navigation meshes define traversable regions for players and NPCs. A NavigationMeshResource stores the mesh geometry, labeled areas, custom flags, and connections between areas. Regions can be assigned traversal costs so pathfinding can prefer faster or easier terrain without necessarily excluding slower regions.

NavigationComponent references the mesh and provides a filter for included or excluded area flags and area costs. NavigationController computes paths from an entity with a navigation component, either synchronously or asynchronously. Path results are returned as nodes, including mesh points and off-mesh connections such as ladders or bridges.

• Define a NavigationMeshResource in Swift or in Reality Composer Pro 3.
• Use NavigationComponent filters to control allowed areas and traversal costs.
• Use NavigationController.computePath to query routes.
• Handle off-mesh connections separately when converting path nodes into character movement.

Query a navigation mesh asynchronously

Use NavigationController to compute a path, then iterate path nodes and distinguish normal mesh points from off-mesh traversal.

extension Entity {
    public func navigate(/* ... */) async {
        let navigator = try! NavigationController(entity: self)

        guard let result = await navigator.computePath(
            from: fromPosition,
            to: toPosition
        ) else { return }

        if result.isEmpty { return }

        for node in result {
            switch node.category {
            case .meshPoint:
                finalPath.append(node.position)
            case .offMeshConnection:
                // handle ladders, bridges, or other special traversal
            }
        }
    }
}

Cloth simulation

RealityKit cloth simulation represents cloth as a mesh whose vertices act as particles and whose edges act as springs. With enough mesh resolution, it can simulate flowing garments, bed covers, curtains, and other deformable surfaces in real time.

A cloth setup uses ClothBodyComponent for the cloth itself, ClothColliderComponent for rigid objects that cloth can collide with, and ClothSimulationComponent to run the simulation. The simulation component owns material references and global simulation settings such as solver choice, gravity, and timestep. Cloth and collider materials expose properties such as spring stiffness and friction.

Pinned cloth can be implemented by selecting mesh vertices near anchor positions and marking them kinematic. Kinematic vertices are moved by entity transforms rather than by the cloth solver, which keeps curtain hoops or other attachment points in place.

• Use ClothBodyComponent with a cloth mesh resource and cloth material properties.
• Use ClothColliderComponent for rigid collision geometry such as furniture or characters.
• Use ClothSimulationComponent on an ancestor to run the simulation for descendant cloth bodies and colliders.
• Pin vertices by selecting them from the cloth mesh and setting their motion type to .kinematic.

Pin cloth vertices to anchor points

Select vertices near each pin entity and make them kinematic so the simulation cannot pull them away from the anchor.

for (pin, pinComponent) in pins {
    let position = pin.position(relativeTo: event.entity)
    let selectionSphere = ClothSphereShape(radius: pinComponent.radius)
    let vertices = clothMesh.vertices(
        in: .sphere(selectionSphere),
        center: position
    )
    clothBody.motionTypes.set(vertexIndices: vertices, value: .kinematic)
}

Performance controls

The session emphasizes using advanced visual and simulation features carefully because they can add rendering or compute cost. Mesh level of detail is the primary optimization shown: render lower-detail geometry when the visual difference is negligible, such as when an object is far away or occupies little screen area.

RealityKit's LevelOfDetailComponent supports switching based on camera distance or screen area. Camera-distance LODs use a maximum distance per level, with the final level commonly using infinity. Screen-area LODs use a minimum fraction of screen area per level.

Apps and games should also monitor thermal state and adapt quality dynamically. When thermal state becomes serious or critical, suggested mitigations include more aggressive LOD switching and lower shadow quality.

• Use LevelOfDetailComponent.addByCameraDistance when distance is a good proxy for visible detail.
• Use LevelOfDetailComponent.addByScreenArea when projected size is a better proxy.
• Use ProcessInfo.thermalStateDidChange to respond to device heat.
• Adapt quality by adjusting LOD thresholds, shadow quality, or other expensive effects.

Create distance-based LODs

Switch from high to lower mesh detail as the entity moves farther from the camera.

let lod0 = [ModelEntity(mesh: lodMesh0)]
let lod1 = [ModelEntity(mesh: lodMesh1)]
let lod2 = [ModelEntity(mesh: lodMesh2)]

let entity = Entity()
LevelOfDetailComponent.addByCameraDistance(to: entity, levels: [
    (entities: lod0, maxDistance: 1.0),
    (entities: lod1, maxDistance: 5.0),
    (entities: lod2, maxDistance: .infinity),
])

React to thermal pressure

Observe thermal-state changes and reduce visual cost when the device reports serious or critical thermal pressure.

NotificationCenter.default.addObserver(
    of: ProcessInfo.self,
    for: .thermalStateDidChange
) { _ in
    switch ProcessInfo.processInfo.thermalState {
    case .nominal, .fair:
        // Stay the course
    case .serious, .critical:
        // Improve performance, for example:
        // - More aggressive LOD switching
        // - Lower shadow quality
    }
}

3D Gaussian splats

RealityKit can render 3D Gaussian splats, a technique for high-quality rendering of volumetric captures from the real world. A capture is represented as many 3D Gaussians, described as ellipsoids with opacity and view-dependent color.

RealityKit does not require a specific Gaussian splat file format. Instead, developers provide buffers for position, scale, rotation, opacity, and spherical harmonics. The spherical harmonics degree controls color variation by view direction; degree 0 represents a solid color from all directions.

The workflow is to create a GaussianSplatResource.BufferResource from the buffers, wrap it in a GaussianSplatResource, create a GaussianSplatComponent, and attach the component to an entity.

• Use Gaussian splats for high-fidelity real-world object or scene captures.
• Provide explicit buffers rather than relying on a RealityKit-specific file format.
• Include spherical harmonics data and degree for view-dependent color.
• Consult the linked "Gaussian splats on visionOS" guide and sample for format conversion and end-to-end setup.

Create and attach a Gaussian splat component

Build a Gaussian splat resource from property buffers, then attach it to an entity through GaussianSplatComponent.

let resource = try GaussianSplatResource.BufferResource(
    count: splatCount,
    position: positionBuffer,
    scale: scaleBuffer,
    rotation: rotationBuffer,
    opacity: opacityBuffer,
    sphericalHarmonics: (sphericalHarmonicsBuffer, degree)
)

let splatResource = GaussianSplatResource(resource)
let splatComponent = GaussianSplatComponent(splatResource)
splatEntity.components.set(splatComponent)

Immersive audio and next steps

RealityKit's custom reverb mesh simulates reflections and reverberation using ray-traced geometrical acoustics. The geometry and material properties of an environment affect how audio sources sound, so a kitchen, museum, or small living room can produce different spatial audio results.

A custom reverb mesh is created with ReverbMeshResource, such as a mesh descriptor, mesh resource, or a simple shoebox. Combine that mesh with preset or custom Audio.Material values to create a simulated Reverb, then attach a ReverbComponent to an entity. Custom materials can define absorption and scattering across frequencies.

The custom reverb mesh feature only works in immersive spaces. In shared spaces, the system uses Apple Vision Pro room-sense reverb geometry built from the real-world surroundings. The session also mentions additional RealityKit updates, including coordinated multi-source audio, high-quality character rendering with subsurface scattering and advanced hair shaders, and portal customizations.

• Use ReverbMeshResource to describe the acoustic geometry of a virtual environment.
• Use preset materials such as .dryWall or define custom Audio.Material absorption and scattering.
• Attach ReverbComponent to activate the simulated reverb in the scene.
• Use custom reverb meshes in immersive spaces; shared spaces use system room-sense reverb geometry.

Create a shoebox reverb mesh

Create a simple inward-facing box reverb mesh and apply a preset drywall acoustic material.

let mesh: ReverbMeshResource = .shoebox(size: [5, 4, 6])
let reverb: Reverb = .simulated(mesh: mesh, materials: [.dryWall])
entity.components.set(ReverbComponent(reverb: reverb))

Define custom reverb materials

Scale a preset material's absorption or build a material from absorption and scattering coefficients.

let thickCarpet: Audio.Material = .carpet.scalingAbsorption { freq in
    0.1
}

let bookshelfAbsorption = Audio.Absorption([
    0.10, 0.15, 0.28, 0.20, 0.15,
    0.10, 0.10, 0.07, 0.07, 0.05
])
let bookshelfScattering = Audio.Scattering([
    500: 0.5,
    1000: 0.6,
    4000: 0.7
])
let bookshelf = Audio.Material(
    absorption: bookshelfAbsorption,
    scattering: bookshelfScattering
)

Resources

• Gaussian splats on visionOS

Related Sessions

• Design no-code games with Reality Composer Pro 3
• Extend Reality Composer Pro 3 functionality with Xcode
• Iterate your spatial scenes faster with Reality Composer Pro 3
• Supercharge your spatial workflows with Reality Composer Pro 3

Unofficial, not associated with Apple. By