Correct way to draw/edit a CVPixelBuffer in Swift in iOS

Question

Is there a standard performant way to edit/draw on a CVImageBuffer/CVPixelBuffer in swift?

All the video editing demos I've found online overlay the drawing (rectangles or text) on the screen and don't directly edit the CVPixelBuffer.

UPDATE I tried using a CGContext but the saved video doesn't show the context drawing

private var adapter: AVAssetWriterInputPixelBufferAdaptor?

extension TrainViewController: CameraFeedManagerDelegate {
    
    func didOutput(sampleBuffer: CMSampleBuffer) {

        let time = CMTime(seconds: timestamp - _time, preferredTimescale: CMTimeScale(600))
        let pixelBuffer: CVPixelBuffer? = CMSampleBufferGetImageBuffer(sampleBuffer)

        guard let context = CGContext(data: CVPixelBufferGetBaseAddress(pixelBuffer),
                                  width: width,
                                  height: height,
                                  bitsPerComponent: 8,
                                  bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
                                  space: colorSpace,
                                  bitmapInfo: alphaInfo.rawValue)
        else {
          return nil
        }

        context.setFillColor(red: 1, green: 0, blue: 0, alpha: 1.0)
        context.fillEllipse(in: CGRect(x: 0, y: 0, width: width, height: height))
        context.flush()

        adapter?.append(pixelBuffer, withPresentationTime: time)


    }
}

Answer 1

You need to call CVPixelBufferLockBaseAddress(pixelBuffer, 0) before creating the bitmap CGContext and CVPixelBufferUnlockBaseAddress(pixelBuffer, 0) after you have finished drawing to the context.

Without locking the pixel buffer, CVPixelBufferGetBaseAddress() returns NULL. This causes your CGContext to allocate new memory to draw into, which is subsequently discarded.

Also double check your colour space. It's easy to mix up your components.

e.g.

guard
    CVPixelBufferLockBaseAddress(pixelBuffer) == kCVReturnSuccess,
    let context = CGContext(data: CVPixelBufferGetBaseAddress(pixelBuffer),
                            width: width,
                            height: height,
                            bitsPerComponent: 8,
                            bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
                            space: colorSpace,
                            bitmapInfo: alphaInfo.rawValue)
else {
    return nil
}

context.setFillColor(red: 1, green: 0, blue: 0, alpha: 1.0)
context.fillEllipse(in: CGRect(x: 0, y: 0, width: width, height: height))

CVPixelBufferUnlockBaseAddress(pixelBuffer)

adapter?.append(pixelBuffer, withPresentationTime: time)