AVCapturePhotoOutput not providing preview buffer

Question

Have a customised camera set up using AVCapturePhotoOutput. Configuring the AVCapturePhotoOutput to provide a preview buffer (thumbnail) in addition to the main JPEG buffer.

Problem is that I'm receiving the preview buffer only once (first capture) and then receiving nil from second time on (the main photoSampleBuffer is received properly always).

Here's how I setup the capture:

func capturePhoto() {

    guard let videoPreviewLayerOrientation = deviceOrientation.videoOrientation else { return }

    sessionQueue.async {
        if let photoOutputConnection = self.photoOutput.connection(withMediaType: AVMediaTypeVideo) {
            photoOutputConnection.videoOrientation = videoPreviewLayerOrientation
        }

        // each photo captured requires a brand new setting object and capture delegate
        let photoSettings = AVCapturePhotoSettings()

        // Capture a JPEG photo with flash set to auto and high resolution photo enabled.
        photoSettings.isHighResolutionPhotoEnabled = true

        //configure to receive a preview image (thumbnail)
        if let previewPixelType = photoSettings.availablePreviewPhotoPixelFormatTypes.first {
            let previewFormat = [kCVPixelBufferPixelFormatTypeKey as String : previewPixelType,
                                 kCVPixelBufferWidthKey as String : NSNumber(value: 160),
                                 kCVPixelBufferHeightKey as String : NSNumber(value: 160)]
            photoSettings.previewPhotoFormat = previewFormat
        }

        // TODO: photoSettings.flashMode = .auto 

        // Use a separate object for the photo capture delegate to isolate each capture life cycle.
        let photoCaptureDelegate = PhotoCaptureDelegate(with: photoSettings, willCapturePhotoAnimation: { [unowned self] in
            // show shutter animation
            self.shutterAnimation()
            }, completed: { [unowned self] (photoCaptureDelegate, photoData, previewThumbnail) in

                self.captureCompleted(photoCaptureDelegate: photoCaptureDelegate, data: photoData, thumbnail: previewThumbnail)
            }
        )
      // The Photo Output keeps a weak reference to the photo capture delegate so we store it in an array
        // to maintain a strong reference to this object until the capture is completed.
      self.inProgressPhotoCaptureDelegates[photoCaptureDelegate.requestedPhotoSettings.uniqueID] = photoCaptureDelegate
        self.photoOutput.capturePhoto(with: photoSettings, delegate: photoCaptureDelegate)
    }
}

In my PhotoCaptureDelegate (which implements AVCapturePhotoCaptureDelegate):

func capture(_ captureOutput: AVCapturePhotoOutput, didFinishProcessingPhotoSampleBuffer photoSampleBuffer: CMSampleBuffer?, previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) {

    if let photoBuffer = photoSampleBuffer {
        photoData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: photoBuffer, previewPhotoSampleBuffer: nil)
    }


    if let previewBuffer = previewPhotoSampleBuffer {
        if let pixelBuffer = CMSampleBufferGetImageBuffer(previewBuffer) {
            photoThumbnail = CIImage(cvPixelBuffer: pixelBuffer)
        }

    }
}

What happens is that first time I capture I receive both the photoSampleBuffer & previewPhotoSampleBuffer. 2nd time and on I only receive photoSampleBuffer and previewPhotoSampleBuffer = nil although when I check resolvedSettings.previewDimensions I get: CMVideoDimensions(width: 160, height: 120)

If I switch camera (front to back) by re configuring the capture session the first capture afterwards is OK and then again no preview buffer. the error param in the delegate callbacks is always nil.

Tested on iPhone 6 running iOS 10.3.1

Answer 1

Found what the solution is but don't fully understand how it causes the initial problem.

I've changed my conversion of the preview sample buffer in the photo capture delegate to convert through a CIContext object to a UIImage. Beforehand I simply created a CIImage and sent it to the UI (different thread) and it seems that the CIImage holds a ref to the original buffer and the UI processing done later on messes with it in some way that effects the next capture (again, don't understand why).

The new code creates a new bitmap copy of the image (through cgImage) and then sends that forward to the UI -> so no processing done on the original buffer.

func capture(_ captureOutput: AVCapturePhotoOutput, didFinishProcessingPhotoSampleBuffer photoSampleBuffer: CMSampleBuffer?, previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) {

    if let photoBuffer = photoSampleBuffer {
        photoData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: photoBuffer, previewPhotoSampleBuffer: nil)
    }

    let previewWidth = Int(resolvedSettings.previewDimensions.width)
    let previewHeight = Int(resolvedSettings.previewDimensions.height)

    if let previewBuffer = previewPhotoSampleBuffer {
        if let imageBuffer = CMSampleBufferGetImageBuffer(previewBuffer) {
            let ciImagePreview = CIImage(cvImageBuffer: imageBuffer)
            let context = CIContext()
            if let cgImagePreview = context.createCGImage(ciImagePreview, from: CGRect(x: 0, y: 0, width:previewWidth , height:previewHeight )) {
                photoThumbnail = UIImage(cgImage: cgImagePreview)
            }
        }
    }
}