Meet the Music Understanding framework

Watch on Apple Developer

TL;DR

What Music Understanding provides

Music Understanding is a framework for on-device musical intelligence across Apple platforms. It handles signal processing and model inference for apps, works offline, and keeps analyzed audio on device.

The framework analyzes six areas: key, rhythm, structure, pace, instrument activity, and loudness. Apple uses it in Final Cut Pro features such as beat detection and music-synchronized montage creation.

• Key identifies the tonic and major/minor mode over time.
• Rhythm returns beats, bars, and an optional global beats-per-minute value.
• Structure returns hierarchical time ranges for sections, segments, and phrases.
• Pace estimates how fast or energetic the music feels over time.
• Instrument activity reports whether instruments are present and how strongly they are active.
• Loudness reports integrated, momentary, short-term, and peak measurements.

Create a session and run analysis

Apps interact with MusicUnderstandingSession. Initialize it with an AVAsset for file-based analysis, or with a custom audio provider for buffer-based input. Calling analyze() requests every supported analysis type by default.

For best performance, request only the analysis types your app needs with the targeted analyze(for:) API. When using targeted analysis, unrequested fields in SessionResult are nil.

• For AVURLAsset input, set AVURLAssetPreferPreciseDurationAndTimingKey to true for the most accurate timing results.
• SessionResult fields are optional because targeted analysis may omit result types.
• All MusicUnderstanding results are Codable, so apps can export or cache analysis results as JSON.

Analyze an imported audio file

Creates an AVURLAsset from a selected audio file, initializes a MusicUnderstandingSession, and awaits all analysis results.

import MusicUnderstanding

.fileImporter(isPresented: $isPresented, allowedContentTypes: [.audio]) { result in
    switch result {
    case .success(let url):
        let asset = AVURLAsset(
            url: url,
            options: [AVURLAssetPreferPreciseDurationAndTimingKey: true]
        )
        let session = try await MusicUnderstandingSession(asset: asset)
        let results = try await session.analyze()
    }
}

SessionResult shape

Each analysis dimension has its own optional result field.

import MusicUnderstanding

public struct SessionResult: Codable, Sendable {
    public let instrumentActivity: InstrumentActivityResult?
    public let key: KeyResult?
    public let loudness: LoudnessResult?
    public let pace: PaceResult?
    public let rhythm: RhythmResult?
    public let structure: StructureResult?
}

Time-based result model

The framework consistently associates musical values with media time. TimedValue attaches a value to a single CMTime, while RangedValue attaches a value to a CMTimeRange.

These types make the results straightforward to render against a playhead, use for beat-synced visuals, align edits to song structure, or drive audio-reactive animations.

• Use TimedValue for sampled values such as loudness or instrument intensity over time.
• Use RangedValue for values that apply across an interval, such as key or pace.
• Structure results use arrays of CMTimeRange for sections, segments, and phrases.

TimedValue and RangedValue

Standard containers used throughout MusicUnderstanding to bind values to media time.

import MusicUnderstanding

public struct TimedValue<Value>: Codable, Equatable, Sendable
where Value: Codable & Equatable & Sendable {
    public let time: CMTime
    public let value: Value
}

public struct RangedValue<Value>: Codable, Equatable, Sendable
where Value: Codable & Equatable & Sendable {
    public let range: CMTimeRange
    public let value: Value
}

Analysis result types

Each result type exposes data at the level needed for common music-aware app features. Rhythm and structure can drive timeline UI and edit alignment; key can support cataloging or DJ workflows; pace and instrument activity can drive visual intensity or automated clip timing.

Loudness uses LUFS for perceived volume. Integrated loudness represents overall loudness, momentary and short-term values are sampled every 100 milliseconds, and peak reports the maximum audio level in decibels.

• RhythmResult includes beat timestamps, bar timestamps, and optional beatsPerMinute. beatsPerMinute can be nil until enough audio has been processed to identify at least two beats.
• StructureResult includes sections, segments, and phrases as hierarchical CMTimeRange arrays.
• InstrumentActivityResult includes coarse presence ranges and detailed per-instrument activity values from 0 to 1.
• LoudnessResult includes integrated, momentary, shortTerm, and peak values.

Rhythm, structure, and pace results

Core timing results for beat grids, structural timelines, and perceived-energy changes.

import MusicUnderstanding

public struct RhythmResult: Codable, Sendable {
    public let beats: [CMTime]
    public let bars: [CMTime]
    public let beatsPerMinute: Float?
}

public struct StructureResult: Codable, Sendable {
    public let sections: [CMTimeRange]
    public let segments: [CMTimeRange]
    public let phrases: [CMTimeRange]
}

public struct PaceResult: Codable, Sendable {
    public let ranges: [MusicUnderstandingSession.RangedValue<Double>]
}

Loudness result

Loudness exposes overall, time-varying, and peak measurements for audio-level visualizations or analysis.

import MusicUnderstanding

public struct LoudnessResult: Codable, Sendable {
    public let integrated: MusicUnderstandingSession.TimedValue<Float>
    public let momentary: [MusicUnderstandingSession.TimedValue<Float>]
    public let shortTerm: [MusicUnderstandingSession.TimedValue<Float>]
    public let peak: MusicUnderstandingSession.TimedValue<Float>
}

Streaming input, loudness, and export workflows

MusicUnderstandingSession also provides a streaming loudness API. LoudnessResult values are delivered through an AsyncSequence for every 100 milliseconds of audio analyzed by the framework.

For custom input, an AudioProvider conforms to AsyncSequence and yields AVReadOnlyAudioPCMBuffer objects. It must yield a final nil to indicate completion.

The Music Understanding Lab sample app exports all analysis data as JSON and demonstrates combining structure with pace to create video clips that start on section boundaries and change duration with the music's energy.

• Use the loudness AsyncSequence when UI needs incremental loudness updates during analysis.
• Use Codable results for JSON export, caching, or precomputing analysis data to bundle with app content.
• For pace-driven video timing, the session suggests deriving clip duration from the pace value so higher-energy sections use shorter clips.

Custom audio provider shape

A buffer-based provider can feed audio into MusicUnderstandingSession instead of using an AVAsset.

import MusicUnderstanding

struct AudioProvider: AsyncSequence, AsyncIteratorProtocol {
    func makeAsyncIterator() -> Self { self }

    mutating func next() async -> AVReadOnlyAudioPCMBuffer? {
        // Return the next audio buffer, or nil to signal completion.
    }
}

Export analysis results as JSON

MusicUnderstanding result types are Codable, enabling JSON export or caching.

import MusicUnderstanding

let session = try await MusicUnderstandingSession(asset: asset)
let results = try await session.analyze()
let encoder = JSONEncoder()
try encoder.encode(results)

Resources

• Creating visuals with Music Understanding analysis results
• MusicUnderstanding

Unofficial, not associated with Apple. By