Running into vImage_Buffer related peak memory issues. When does vImage_Buffer.free() actually free memory?

Question

I am working on a project where the UI has to generate previews for many larger still images. I am using a C++ library to apply various lookup tables and conversions to the images which requires the image to be in a RGB float buffer. I use Accelerate to convert the source image into Float and back into a RGB image. I am managing conversion jobs through a serial DispatchQueue so that one conversion is done after another.

What I am running into is peak memory issues. Despite the fact that I manage jobs sequentially and processing images one by one I pile up memory in the GB range. If I have enough memory to complete all processing the memory gets freed and nothing leaks - but during processing I am using up multiple times more memory than I should.

In instruments I have checked that I stack up vImage_Buffer instances (every operation creates 3 but I see many) each multiple MB in size - despite calling free() on the buffers in the function at the end.

I am rather clueless atm about why it piles up memory? My working theory is that vImage_Buffer.free() does not free memory immidiately? But needs something to happen to actually free the memory (it seems like it frees it when all the processing is done, but these are independent jobs really and there is no common thing that says I'm done.). I am actually quite puzzled. How can I limit the amount of memory I consume? I have thought about keeping the buffers around - but the problem is that the images do not have the same size.

It is the first time I use Accelerate and vImage - what did I miss?

Any insight would be much appreciated....

Here is the function (yes I do call free()):

@available(iOS 16.0,*)
static func processImage(from cgImage:CGImage, lookData:Data) throws -> CGImage {
    
    let width = UInt32(cgImage.width)
    let height = UInt32(cgImage.height)
    
    let Rec709 = CGColorSpace(name: CGColorSpace.itur_709)!
            
    let bitmapInfo_3ChanFloat:CGBitmapInfo = CGBitmapInfo(
        rawValue:CGBitmapInfo.floatComponents.rawValue | CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.none.rawValue
    )
    
    var src_format = vImage_CGImageFormat(
        bitsPerComponent: cgImage.bitsPerComponent,
        bitsPerPixel: cgImage.bitsPerPixel,
        colorSpace: cgImage.colorSpace!,
        bitmapInfo: cgImage.bitmapInfo)!
    
    var convertible_format = vImage_CGImageFormat(
        bitsPerComponent: 32,
        bitsPerPixel: 3 * 32,
        colorSpace: cgImage.colorSpace!,
        bitmapInfo: bitmapInfo_3ChanFloat)!
    
    var converted_format = vImage_CGImageFormat(
        bitsPerComponent: 32,
        bitsPerPixel: 3 * 32,
        colorSpace: Rec709,
        bitmapInfo: bitmapInfo_3ChanFloat)!
    
    var end_format = vImage_CGImageFormat(
        bitsPerComponent: 8,
        bitsPerPixel: 4 * 8,
        colorSpace: Rec709,
        bitmapInfo: .init(rawValue: CGImageAlphaInfo.noneSkipLast.rawValue))!
    
    let to_converter = vImageConverter_CreateWithCGImageFormat(
        &src_format,
        &convertible_format,
        nil,
        vImage_Flags(kvImagePrintDiagnosticsToConsole),
        nil).takeRetainedValue()
    
    let back_converter = vImageConverter_CreateWithCGImageFormat(
        &converted_format,
        &end_format,
        nil,
        vImage_Flags(kvImagePrintDiagnosticsToConsole),
        nil).takeRetainedValue()
    
    var src_buffer = try vImage_Buffer(cgImage: cgImage)
    
    var conversion_buffer = vImage_Buffer()
    vImageBuffer_Init(
      &conversion_buffer,
      UInt(height),
      UInt(width),
      convertible_format.bitsPerPixel,
      vImage_Flags(kvImagePrintDiagnosticsToConsole)
    )
    
    var end_buffer = vImage_Buffer()
    vImageBuffer_Init(
      &end_buffer,
      UInt(height),
      UInt(width),
      end_format.bitsPerPixel,
      vImage_Flags(kvImagePrintDiagnosticsToConsole)
    )
    
    vImageConvert_AnyToAny(
      to_converter,
      &src_buffer,
      &conversion_buffer,
      nil,
      vImage_Flags(kvImagePrintDiagnosticsToConsole)
    )

    let imageBuffer = conversion_buffer.data.assumingMemoryBound(to: Float.self)
    let size = Int(width * height * 4 * 3)
    
    try lookData.withUnsafeBytes { (rawLookBuffer:UnsafeRawBufferPointer) in
        let lookBuffer:UnsafePointer<UInt8> = rawLookBuffer.baseAddress!.assumingMemoryBound(to: UInt8.self)
        
        let result = applyLook_CPP(imageBuffer, size, width, height, lookBuffer, rawLookBuffer.count)
        if result < 0 {
            throw Errors.ImageSDKFailed(error: result)
        }
    }
    
    vImageConvert_AnyToAny(
      back_converter,
      &conversion_buffer,
      &end_buffer,
      nil,
      vImage_Flags(kvImagePrintDiagnosticsToConsole)
    )
    
    let result = try end_buffer.createCGImage(format: end_format)
    
    src_buffer.free()
    conversion_buffer.free()
    end_buffer.free()
    
    return result
}

Answer 1

So the solution to my peak memory problem is called: autoreleasepool! I discovered it accidentially while digging through the Debug Memory Graph...

Wrapping the processing into an autoreleasepool{ } block fixed the memory issues and the memory was freed more frequently.

The memory issue wasn't caused by the vImage buffers themselves, but the cgImages referencing them. (cgImage is some old code it seems). So if you run into this type of problem make sure that the autorelease covers the entire lifespan of the cgImages (e.g. if you save them to disk) otherwise it won't be effective.

You never stop learning :-)