Use foveated streaming to bring immersive content to visionOS

Watch on Apple Developer

TL;DR

What Foveated Streaming provides

Foveated Streaming is a visionOS framework for bringing externally rendered immersive content, including OpenXR applications, to Apple Vision Pro. A visionOS receiver app connects to a streaming endpoint such as a Windows PC or cloud machine, sends input data such as hands, controller positions, and microphone, and receives video and audio from the OpenXR app.

The stream is optimized using Apple Vision Pro eye tracking: content near the person's focus is streamed in higher detail. visionOS includes NVIDIA CloudXR streaming support, allowing high-performance wireless streaming over Wi‑Fi from a local PC or from the cloud.

The intended architecture combines a native visionOS app with an OpenXR host app. The visionOS side can use SwiftUI, ARKit, and RealityKit, while the endpoint uses the Foveated Streaming Protocol and the NVIDIA CloudXR OpenXR runtime.

• Introduced in visionOS 26.4 as the FoveatedStreaming framework.
• Targets immersive OpenXR content rendered on an external endpoint.
• Supports native visionOS UI and spatial features around the streamed content.
• Uses foveated video processing while preserving the high-fidelity area around the viewer's gaze.

Receiver app on visionOS

The visionOS receiver app is responsible for welcoming users, discovering and pairing with endpoints, and presenting the streamed content. The core API is FoveatedStreamingSession. Calling connect() presents available endpoints, handles pairing UI, and returns once the session is connected and ready to stream.

Pairing requires the endpoint to display a QR code containing pairing information. The framework presents scanning UI automatically, and the person scans the code by looking at it. After connection, pass the session to an ImmersiveSpace to include the streamed content in the spatial scene.

Because this is still a SwiftUI app, the receiver can include normal windows, volumetric windows, controls to pause/resume, widgets in the immersive space, spatial gestures, and immersion styles such as progressive immersion.

• Create one FoveatedStreamingSession for the app's streaming lifecycle.
• Use connect() from Swift concurrency to initiate endpoint selection and pairing.
• Use ImmersiveSpace(foveatedStreaming:) to display the stream.
• Compose additional SwiftUI content in windows or inside the immersive space.

Connect to a streaming endpoint

A SwiftUI control can initiate discovery, pairing, and connection by calling connect() on the session.

import SwiftUI
import FoveatedStreaming

struct ConnectView: View {
    let session: FoveatedStreamingSession

    var body: some View {
        Button("Connect") {
            Task {
                try await session.connect()
            }
        }
    }
}

Display a Foveated Streaming session in an ImmersiveSpace

Passing the session to ImmersiveSpace includes the streamed OpenXR content in the immersive scene.

import SwiftUI
import FoveatedStreaming

@main
struct FoveatedStreamingSampleApp: App {
    private let session = FoveatedStreamingSession()

    var body: some SwiftUI.Scene {
        ImmersiveSpace(foveatedStreaming: session)
    }
}

Streaming endpoint and protocol integration

The streaming endpoint has two main responsibilities: implement Apple's Foveated Streaming Protocol and use the OpenXR runtime provided by the NVIDIA CloudXR SDK. Apple provides open-source Windows sample code on GitHub with a reference implementation of the protocol, an example OpenXR application, and setup guidance for the CloudXR runtime.

The Foveated Streaming Protocol is a lightweight TCP-based connection separate from the media stream. It authenticates the secure foveated stream and communicates session state between visionOS and the endpoint. Endpoints are made visible on the local network with Bonjour, visionOS connects after endpoint selection, pairing occurs if needed, and the stream begins when the endpoint reports that content is ready.

Protocol messages are JSON-encoded and follow a request/acknowledge pattern. During barcode pairing, the barcode is also JSON and contains a client token and a hash of the secure connection's certificate, both provided by the NVIDIA CloudXR SDK. Endpoint implementations should monitor session status, including pause behavior when Apple Vision Pro goes to sleep, and remain available so the user can reconnect.

• Use Bonjour for local endpoint visibility.
• Use Apple's Foveated Streaming Protocol for authentication, pairing, and session state.
• Use NVIDIA CloudXR's Windows OpenXR runtime for streaming transport.
• Support reconnection by keeping the endpoint available after sleep or connection loss.

CloudXR OpenXR behavior and content composition

NVIDIA CloudXR provides the OpenXR runtime for Windows, so an OpenXR application connects to that runtime and CloudXR handles the streaming details. visionOS input data is provided to OpenXR automatically, including hand tracking via OpenXR extensions. PlayStation VR2 Sense Controllers can also pass through.

For better mixed immersion, the session recommends providing a depth buffer and an alpha channel. Depth enables streamed OpenXR content and native rendered content to occlude each other appropriately, while alpha allows the stream to blend with the user's surroundings.

Native content can be composed with the stream directly in the same immersive space. Adding SwiftUI content or RealityKit content lets the receiver app provide controls, widgets, overlays, and on-device 3D rendering around the remote application.

• Use the OpenXR hand tracking extension where appropriate.
• Provide a depth buffer for better occlusion behavior.
• Provide an alpha channel to mix streamed content with the physical environment.
• Add RealityKit content with RealityView inside the foveated streaming ImmersiveSpace.

Compose SwiftUI content with Foveated Streaming

The receiver can combine streamed content with SwiftUI windows and spatial controls.

import SwiftUI
import FoveatedStreaming

@main
struct FoveatedStreamingSampleApp: App {
    private let session = FoveatedStreamingSession()
    private let appModel = AppModel()

    var body: some SwiftUI.Scene {
        Window("Main", id: appModel.mainWindowId) {
            ContentView(session: session)
                .environment(appModel)
                .environment(session)
        }

        ImmersiveSpace(foveatedStreaming: session) {
            SpatialContainer {
                ReopenMainWindowView().environment(appModel)
                TransformStreamWidgetView().environment(session)
            }
        }
    }
}

Compose RealityKit content with Foveated Streaming

RealityKit content can be rendered natively alongside the streamed OpenXR scene.

import SwiftUI
import RealityKit
import FoveatedStreaming

@main
struct FoveatedStreamingSampleApp: App {
    private let session = FoveatedStreamingSession()
    private let appModel = AppModel()

    var body: some SwiftUI.Scene {
        ImmersiveSpace(foveatedStreaming: session) {
            RealityView { content in
                // Add native RealityKit content.
            }
        }
    }
}

Enhancing and measuring a streamed experience

A foveated streaming experience is split across two apps: the visionOS receiver and the host app on the endpoint. FoveatedStreamingSession exposes message channels on visionOS, and CloudXR provides an OpenXR extension for message channels on the endpoint. Messages are opaque data blobs, so the apps define their own payloads.

Message channels can drive app-level coordination, such as a SwiftUI level picker sending commands to the OpenXR app and the endpoint reporting loading progress. They can also transmit ARKit-derived alignment data so the remote scene matches the user's physical space. The framework also provides API support for converting between OpenXR and ARKit coordinate frames.

For performance work, Xcode includes a Foveated Streaming instrument. It reports stream metrics such as bandwidth, pose latency, frame rate, and related statistics to help diagnose problems before shipping.

• Use message channels for app-specific commands and progress reporting.
• Use ARKit for physical-world alignment and send relevant data to the endpoint when needed.
• Use coordinate-frame conversion between OpenXR and ARKit through FoveatedStreamingSession.
• Profile bandwidth, pose latency, and frame rate with the Foveated Streaming instrument in Xcode.

Resources

• Analyzing the performance of a foveated streaming session
• Establishing foveated streaming sessions with Apple Vision Pro
• Streaming a CloudXR application to Apple Vision Pro with foveation
• Creating a foveated streaming client on visionOS
• Foveated Streaming
• StreamingSession: Streaming immersive content from a CloudXR™ application to visionOS and iOS

Related Sessions

• Build next-generation experiences with visionOS 27
• visionOS Group Lab

Unofficial, not associated with Apple. By