How does one obtain image buffers in their original format from a video stream using AVFoundation?

Question

In Apple's documentation for AVAssetReaderTrackOutput, it indicates the following about the parameter for outputSettings when instantiating an instance using +[AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:outputSettings:]

A value of nil configures the output to vend samples in their original format as stored by the specified track.

When using it on e.g. an MP4 video asset, it will seemingly step through frames in decode order (i.e. out-of-order with respect to display), however all queries to delivered CMSampleBufferRef objects using CMSampleBufferGetImageBuffer yields NULL CVImageBufferRef objects.

The only way I can ensure delivery of image buffer objects is to provide a pixel buffer format to outputSettings:, such as kCVPixelFormatType_32ARGB for the kCVPixelBufferPixelFormatTypeKey dictionary entry.

Another interesting side-effect of doing this, is that frames are then delivered in display order, and the underlying decode-order of frames is abstracted/hidden away.

Any ideas why this is so?

Answer 1

Like you I expected that setting an outputSettings of nil would result in output of native format video frames but this is not the case, you must specify something in order to get a valid CVSampleBufferRef.

All is not lost, using a "barely there" dictionary seems to output frames in their native format,

AVAsset asset = [AVURLAsset URLAssetWithURL:inputURL options:nil];
AVAssetTrack *videoTrack = [[asset tracksWithMediaCharacteristic:AVMediaCharacteristicVisual] objectAtIndex:0];

NSDictionary *decompressionSettings =
     @{ (id)kCVPixelBufferIOSurfacePropertiesKey : [NSDictionary dictionary] };
AVAssetReaderTrackOutput trackOutput = [[AVAssetReaderTrackOutput alloc] initWithTrack:videoTrack outputSettings:decompressionSettings];
...

IOSurfaceOptions are simply default - further reading for reference: https://developer.apple.com/documentation/corevideo/kcvpixelbufferiosurfacepropertieskey?language=objc