WWDC 2026: Build live production tools for Apple Immersive Video (Session 338)

Build Apple Immersive Video live production tools using ProRes, ASAF audio, per-frame metadata, SMPTE 2110, AVFoundation, VideoToolbox, and IMS.

Watch on Apple Developer

TL;DR

Apple Immersive Video live production uses streamed ProRes video, uncompressed PCM ASAF audio, and per-frame JSON metadata as the common interchange format between tools.
Real-time transport is based on SMPTE 2110 over IP: ProRes over 2110-22, ASAF audio over 2110-30, and user-defined metadata over 2110-41.
Recording and replay workflows should preserve quality by copying the native ProRes payload into QuickTime MOV files instead of decode/re-encode cycles.
Use AVAssetWriter, VideoToolbox projection metadata, and Immersive Media Support to write video, audio, and synchronized immersive metadata tracks for editorial, replay, and playout.

Live Apple Immersive Video production follows the same broad structure as traditional broadcast: a production domain captures and creatively combines media, then a delivery domain encodes and streams the result to viewers. Production tools can range from large broadcast trucks and studios to smaller venues, but they share core concepts: cameras, graphics, replay, switching, microphones, audio mixing, and routing.

The session focuses on the production domain: how devices exchange live immersive media and how developers can build tools that participate in that workflow.

Live cameras capture multiple viewpoints or angles.
Graphics systems generate overlays such as lower thirds, scoreboards, and animations.
Replay systems record media for instant replay, archive, editorial, or post-production use.
Video switchers assemble cameras and graphics into the final creative output.
Audio consoles combine microphones, commentary, venue sound, and other sources into the final mix.
A media router connects devices so signals can be exchanged across the production workflow.

What changes for immersive live

Immersive live production must preserve fidelity and presence throughout the pipeline because the viewer is placed inside the event rather than watching a flat broadcast. That changes the scale of the media and the requirements for each production device.

Compared with typical 2D broadcast production, Apple Immersive Video uses much larger video resolution and higher frame rate, and the audio mix is far richer than stereo or 5.1 surround. These requirements exceed what many traditional broadcast formats, transport methods, and tools were designed to handle.

Video resolution is described as 32 times larger than typical 2D broadcast production.
The format is produced at twice the frame rate referenced for typical 2D broadcast production.
Apple Spatial Audio Format (ASAF) mixes can contain 64 or more channels.
Tools need to preserve high-fidelity video, spatial audio, and synchronized metadata through capture, routing, recording, editing, replay, and playout.

Immersive live media format

The live immersive production format combines three existing media approaches into a shared language that production devices can exchange: streamed ProRes frames for video, uncompressed PCM tracks for spatial audio, and per-frame JSON metadata. Devices that output, ingest, or both output and ingest media need to comply with these media types to interoperate in the wider ecosystem.

ProRes is used instead of uncompressed video frames because it provides a practical bandwidth and processing profile while preserving the image quality required for immersive production. Apple Silicon is optimized for ProRes processing, making it a strong platform for production tools.

Apple Immersive Live Video is composed of streamed ProRes frames.
ASAF audio is carried as standard uncompressed PCM audio tracks containing high-order ambisonic beds and spatial audio objects.
Metadata is delivered as per-frame JSON objects describing related video and audio attributes such as lens calibrations, creative events, spatial audio behavior, and motion-related data.
The format is intended for real-time tool interoperability across cameras, graphics systems, monitors, switchers, encoders, recorders, and replay systems.

Real-time transport over SMPTE 2110

Live immersive feeds are transported between production devices as individual SMPTE 2110 media streams over IP. SMPTE 2110 uses multicast RTP to carry timing information, user flags, metadata, and the primary media payload, and is widely used in professional broadcast facilities.

Each stream carries one media class, with the transport behavior defined by a sub-standard within the broader SMPTE 2110 family.

Immersive ProRes video uses SMPTE 2110-22, the standard for compressed media over IP.
A 2110-22 immersive video flow contains both left-eye and right-eye essences inside a single stream, avoiding side-by-side frame packing and avoiding separate IP streams per eye.
ASAF audio uses SMPTE 2110-30 streams carrying the ambisonic and object channels that make up the spatial audio mix.
Per-frame JSON metadata uses SMPTE 2110-41, the user-defined metadata transport over IP.
Using separate 2110 flows for video, audio, and metadata keeps the production architecture aligned with professional IP media workflows.

Recording, MOV files, and playback

Recording and replay are central to live production, but immersive workflows cannot tolerate unnecessary generational loss from repeated encode/decode/re-encode cycles. The live immersive format avoids this because the video payload is already ProRes and file-friendly: the same ProRes frames can be copied directly into QuickTime MOV files and later read back for playout into SMPTE 2110 streams.

AVFoundation's AVAssetWriter is used to write MOV tracks. Video goes into QuickTime MOV video tracks, uncompressed PCM audio goes into audio tracks, and JSON-derived immersive metadata is written into MEBX tracks after parsing and conversion with Immersive Media Support.

Use AVAssetWriter to save ProRes video feeds into MOV without additional encoding or decoding.
Set the VideoToolbox projection kind so the MOV gets the correct vexu static metadata that identifies it as Apple Immersive Video.
Write ASAF audio as uncompressed PCM audio tracks in the MOV container.
Deserialize and parse streamed JSON metadata, then use Immersive Media Support to create lens calibration objects, camera IDs, and other synchronized metadata objects.
For playback, read video, audio, and metadata tracks from MOV and retransmit them into SMPTE 2110 output streams for use in the live production workflow.

Set compression properties for Apple Immersive Video `vexu` metadata

When writing the MOV video track, set kVTProjectionKind_AppleImmersiveVideo in the compression properties so the file is signaled as Apple Immersive Video to other applications.

import VideoToolbox

let compressionProperties: [String: Any] = [
    // ...
    kVTCompressionPropertyKey_ProjectionKind as String: kVTProjectionKind_AppleImmersiveVideo
    // ...
]

Build live production tools for Apple Immersive Video

TL;DR

Live production context