Deep Copy of Audio CMSampleBuffer

Question

I am trying to create a copy of a CMSampleBuffer as returned by captureOutput in a AVCaptureAudioDataOutputSampleBufferDelegate.

The problem I am having is that my frames coming from delegate method captureOutput:didOutputSampleBuffer:fromConnection: being dropped after I retain them in CFArray for long time.

Obviously, I need to create deep copies of incoming buffers for further processing. I also know that CMSampleBufferCreateCopy only creates shallow copies.

There are few related questions were asked on SO:

• Pulling data from a CMSampleBuffer in order to create a deep copy
• Creating copy of CMSampleBuffer in Swift returns OSStatus -12743 (Invalid Media Format)
• Deep Copy of CMImageBuffer or CVImageBuffer

But none of them helps me to use correctly CMSampleBufferCreate function with 12 parameters:

  CMSampleBufferRef copyBuffer;

  CMBlockBufferRef data = CMSampleBufferGetDataBuffer(sampleBuffer);
  CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer);
  CMItemCount itemCount = CMSampleBufferGetNumSamples(sampleBuffer);

  CMTime duration = CMSampleBufferGetDuration(sampleBuffer);
  CMTime presentationStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
  CMSampleTimingInfo timingInfo;
  timingInfo.duration = duration;
  timingInfo.presentationTimeStamp = presentationStamp;
  timingInfo.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer);


  size_t sampleSize = CMBlockBufferGetDataLength(data);
  CMBlockBufferRef sampleData;

  if (CMBlockBufferCopyDataBytes(data, 0, sampleSize, &sampleData) != kCMBlockBufferNoErr) {
    VLog(@"error during copying sample buffer");
  }

  // Here I tried data and sampleData CMBlockBuffer instance, but no success
  OSStatus status = CMSampleBufferCreate(kCFAllocatorDefault, data, isDataReady, nil, nil, formatDescription, itemCount, 1, &timingInfo, 1, &sampleSize, &copyBuffer);

  if (!self.sampleBufferArray)  {
    self.sampleBufferArray = CFArrayCreateMutable(NULL, 0, &kCFTypeArrayCallBacks);
    //EXC_BAD_ACCESS crash when trying to add sampleBuffer to the array
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  } else  {
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  }

How do you deep copy Audio CMSampleBuffer? Feel free to use any language (swift/objective-c) in your answers.

Answer 1

Here is a working solution I finally implemented. I sent this snippet to Apple Developer Technical support and asked them to check if it is a correct way to copy incoming sample buffer. The basic idea is copy AudioBufferList and then create a CMSampleBuffer and set AudioBufferList to this sample.

AudioBufferList audioBufferList;
CMBlockBufferRef blockBuffer;
//Create an AudioBufferList containing the data from the CMSampleBuffer,
//and a CMBlockBuffer which references the data in that AudioBufferList.
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer);
NSUInteger size = sizeof(audioBufferList);
char buffer[size];

memcpy(buffer, &audioBufferList, size);
//This is the Audio data.
NSData *bufferData = [NSData dataWithBytes:buffer length:size];

const void *copyBufferData = [bufferData bytes];
copyBufferData = (char *)copyBufferData;

CMSampleBufferRef copyBuffer = NULL;
OSStatus status = -1;

/* Format Description */

AudioStreamBasicDescription audioFormat = *CMAudioFormatDescriptionGetStreamBasicDescription((CMAudioFormatDescriptionRef) CMSampleBufferGetFormatDescription(sampleBuffer));

CMFormatDescriptionRef format = NULL;
status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &audioFormat, 0, nil, 0, nil, nil, &format);

CMFormatDescriptionRef formatdes = NULL;
status = CMFormatDescriptionCreate(NULL, kCMMediaType_Audio, 'lpcm', NULL, &formatdes);
if (status != noErr)
{
  NSLog(@"Error in CMAudioFormatDescriptionCreator");
  CFRelease(blockBuffer);
  return;
}

/* Create sample Buffer */
CMItemCount framesCount = CMSampleBufferGetNumSamples(sampleBuffer);
CMSampleTimingInfo timing   = {.duration= CMTimeMake(1, 44100), .presentationTimeStamp= CMSampleBufferGetPresentationTimeStamp(sampleBuffer), .decodeTimeStamp= CMSampleBufferGetDecodeTimeStamp(sampleBuffer)};

status = CMSampleBufferCreate(kCFAllocatorDefault, nil , NO,nil,nil,format, framesCount, 1, &timing, 0, nil, &copyBuffer);

if( status != noErr) {
  NSLog(@"Error in CMSampleBufferCreate");
  CFRelease(blockBuffer);
  return;
}

/* Copy BufferList to Sample Buffer */
AudioBufferList receivedAudioBufferList;
memcpy(&receivedAudioBufferList, copyBufferData, sizeof(receivedAudioBufferList));

//Creates a CMBlockBuffer containing a copy of the data from the
//AudioBufferList.
status = CMSampleBufferSetDataBufferFromAudioBufferList(copyBuffer, kCFAllocatorDefault , kCFAllocatorDefault, 0, &receivedAudioBufferList);
if (status != noErr) {
  NSLog(@"Error in CMSampleBufferSetDataBufferFromAudioBufferList");
  CFRelease(blockBuffer);
  return;
}

Code-Level Support answer:

This code looks ok (though you’ll want to add some additional error checking). I've successfully tested it in an app that implements the AVCaptureAudioDataOutput delegate captureOutput:didOutputSampleBuffer:fromConnection: method to capture and record audio. The captured audio I'm getting when using this deep copy code appears to be the same as what I get when directly using the provided sample buffer (without the deep copy).

Apple Developer Technical Support

Answer 2

Couldn't find a decent answer doing this in Swift. Here's an extension:

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        guard let formatDesc = CMSampleBufferGetFormatDescription(self),
              let data = try? self.dataBuffer?.dataBytes() else {
                  return nil
              }
        let nFrames = CMSampleBufferGetNumSamples(self)
        let pts = CMSampleBufferGetPresentationTimeStamp(self)
        let dataBuffer = data.withUnsafeBytes { (buffer) -> CMBlockBuffer? in
            var blockBuffer: CMBlockBuffer?
            let length: Int = data.count
            guard CMBlockBufferCreateWithMemoryBlock(
                allocator: kCFAllocatorDefault,
                memoryBlock: nil,
                blockLength: length,
                blockAllocator: nil,
                customBlockSource: nil,
                offsetToData: 0,
                dataLength: length,
                flags: 0,
                blockBufferOut: &blockBuffer) == noErr else {
                    print("Failed to create block")
                    return nil
                }
            guard CMBlockBufferReplaceDataBytes(
                with: buffer.baseAddress!,
                blockBuffer: blockBuffer!,
                offsetIntoDestination: 0,
                dataLength: length) == noErr else {
                    print("Failed to move bytes for block")
                    return nil
                }
            return blockBuffer
        }
        guard let dataBuffer = dataBuffer else {
            return nil
        }
        var newSampleBuffer: CMSampleBuffer?
        CMAudioSampleBufferCreateReadyWithPacketDescriptions(
            allocator: kCFAllocatorDefault,
            dataBuffer: dataBuffer,
            formatDescription: formatDesc,
            sampleCount: nFrames,
            presentationTimeStamp: pts,
            packetDescriptions: nil,
            sampleBufferOut: &newSampleBuffer
        )
        return newSampleBuffer
    }
}

Answer 3

LLooggaann's solution is simpler and works well, however, in case anyone is interested, I migrated the original solution to Swift 5.6:

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        var audioBufferList : AudioBufferList = AudioBufferList()
        var blockBuffer : CMBlockBuffer?

        let sizeOfAudioBufferList = MemoryLayout<AudioBufferList>.size
        
        //Create an AudioBufferList containing the data from the CMSampleBuffer.
        CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(self,
                                                                bufferListSizeNeededOut: nil,
                                                                bufferListOut: &audioBufferList,
                                                                bufferListSize: sizeOfAudioBufferList,
                                                                blockBufferAllocator: nil,
                                                                blockBufferMemoryAllocator: nil,
                                                                flags: 0,
                                                                blockBufferOut: &blockBuffer)

        guard audioBufferList.mNumberBuffers == 1 else { return nil }  //TODO: Make this generic for any number of buffers
        
        /* Deep copy the audio buffer */
        let audioBufferDataSize = Int(audioBufferList.mBuffers.mDataByteSize)
        let audioBuffer = audioBufferList.mBuffers
        let audioBufferDataCopyPointer = UnsafeMutableRawPointer.allocate(byteCount: audioBufferDataSize, alignment: 1)
                
        defer {
            audioBufferDataCopyPointer.deallocate()
        }
        
        memcpy(audioBufferDataCopyPointer, audioBufferList.mBuffers.mData, audioBufferDataSize)
        
        let copiedAudioBuffer = AudioBuffer(mNumberChannels: audioBuffer.mNumberChannels,
                                            mDataByteSize: audioBufferList.mBuffers.mDataByteSize,
                                            mData: audioBufferDataCopyPointer)
        
        /* Create a new audio buffer list with the deep copied audio buffer */
        var copiedAudioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: copiedAudioBuffer)

        /* Copy audio format description, to be used in the new sample buffer */
        guard let sampleBufferFormatDescription = CMSampleBufferGetFormatDescription(self) else { return nil }

        /* Create copy of timing for new sample buffer */
        var duration = CMSampleBufferGetDuration(self)
        duration.value /= Int64(numSamples)
        var timing = CMSampleTimingInfo(duration: duration,
                                        presentationTimeStamp: CMSampleBufferGetPresentationTimeStamp(self),
                                        decodeTimeStamp: CMSampleBufferGetDecodeTimeStamp(self))

        /* New sample buffer preparation, using the audio format description, and the timing information. */
        let sampleCount = CMSampleBufferGetNumSamples(self)
        var newSampleBuffer : CMSampleBuffer?

        guard CMSampleBufferCreate(allocator: kCFAllocatorDefault,
                                   dataBuffer: nil,
                                   dataReady: false,
                                   makeDataReadyCallback: nil,
                                   refcon: nil,
                                   formatDescription: sampleBufferFormatDescription,
                                   sampleCount: sampleCount,
                                   sampleTimingEntryCount: 1,
                                   sampleTimingArray: &timing,
                                   sampleSizeEntryCount: 0,
                                   sampleSizeArray: nil,
                                   sampleBufferOut: &newSampleBuffer) == noErr else { return nil }

        //Create a CMBlockBuffer containing a copy of the data from the AudioBufferList, add to new sample buffer.
        let status = CMSampleBufferSetDataBufferFromAudioBufferList(newSampleBuffer!,
                                                                    blockBufferAllocator: kCFAllocatorDefault,
                                                                    blockBufferMemoryAllocator: kCFAllocatorDefault,
                                                                    flags: 0,
                                                                    bufferList: &copiedAudioBufferList)
        
        guard status == noErr else { return nil }

        return newSampleBuffer
    }
}

Answer 4

The earlier posted answers are great! I've used LLooggaann's answer succesfully in my codebase and refactored it with modern Swift APIs. This gives much shorter code that also throws errors in case something goes wrong and has a lot less pointer & memory management going on.

For all future devs needing this and wanting a more modern compact implementation:

extension CMSampleBuffer {
    struct InvalidAudioSampleBuffer: Swift.Error {}
    
    func deepCopyAudioSampleBuffer() throws -> CMSampleBuffer {
        guard let formatDescription, let dataBuffer else { throw InvalidAudioSampleBuffer() }

        let data = try dataBuffer.dataBytes()
        let dataBufferCopy = try data.withUnsafeBytes { buffer -> CMBlockBuffer in
            let blockBuffer = try CMBlockBuffer(length: data.count)
            try blockBuffer.replaceDataBytes(with: buffer)
            return blockBuffer
        }

        return try CMSampleBuffer(dataBuffer: dataBufferCopy,
                                  formatDescription: formatDescription,
                                  numSamples: numSamples,
                                  presentationTimeStamp: presentationTimeStamp,
                                  packetDescriptions: [])
    }
}