How to create depth data and add it to an image?

Question

Sorry, I duplicated this question How to build AVDepthData manually, because it doesn't have answers I want and I don't have enough rep to comment there. If you don't mind, I could remove my question in the future and ask somebody to move future answers to that topic.

So, my goal is to create depth data and attach it to an arbitrary image. There is an article on how to do it https://developer.apple.com/documentation/avfoundation/avdepthdata/creating_auxiliary_depth_data_manually, but I don't know how to implement any step of it. I won't post all questions at once and start with the first one.

As a first step a depth image must be converted per-pixel from grayscale to depth or disparity values. I took this snippet from the aforementioned topic:

func buildDepth(image: UIImage) -> AVDepthData? {
        let width = Int(image.size.width)
        let height = Int(image.size.height)
        var maybeDepthMapPixelBuffer: CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_DisparityFloat32, nil, &maybeDepthMapPixelBuffer)

        guard status == kCVReturnSuccess, let depthMapPixelBuffer = maybeDepthMapPixelBuffer else {
            return nil
        }

        CVPixelBufferLockBaseAddress(depthMapPixelBuffer, .init(rawValue: 0))

        guard let baseAddress = CVPixelBufferGetBaseAddress(depthMapPixelBuffer) else {
            return nil
        }

        let buffer = unsafeBitCast(baseAddress, to: UnsafeMutablePointer<Float32>.self)

        for i in 0..<width * height {
            buffer[i] = 0 // disparity must be calculated somehow, but set to 0 for testing purposes
        }

        CVPixelBufferUnlockBaseAddress(depthMapPixelBuffer, .init(rawValue: 0))

        let info: [AnyHashable: Any] = [kCGImagePropertyPixelFormat: kCVPixelFormatType_DisparityFloat32,
                                        kCGImagePropertyWidth: image.size.width,
                                        kCGImagePropertyHeight: image.size.height,
                                        kCGImagePropertyBytesPerRow: CVPixelBufferGetBytesPerRow(depthMapPixelBuffer)]

        let metadata = generateMetadata(image: image)
        let dic: [AnyHashable: Any] = [kCGImageAuxiliaryDataInfoDataDescription: info,
// I get an error when converting baseAddress to CFData
                                       kCGImageAuxiliaryDataInfoData: baseAddress as! CFData,
                                       kCGImageAuxiliaryDataInfoMetadata: metadata]

        guard let depthData = try? AVDepthData(fromDictionaryRepresentation: dic) else {
            return nil
        }

        return depthData
    }

Then the article says to load a base address of a pixel buffer (in which is the disparity map) as CFData and pass it as kCGImageAuxiliaryDataInfoData value into a CFDictionary. But I get an error when converting baseAddress to CFData. I tried to convert the pixel buffer itself too, but without luck. What do I have to pass as kCGImageAuxiliaryDataInfoData? Did I create the disparity buffer correctly in the first place?

Aside from this problem it would be cool if somebody could direct me to some sample code on how to do the whole thing.

Answer 1

Your question really helped me get from cvPixelBuffer to AVDepthData so thank you. It was about 95% of the way there.

To fix your (and mine) issue I added the following:

let bytesPerRow = CVPixelBufferGetBytesPerRow(depthMapPixelBuffer)
let size = bytesPerRow * height;
... code code code ...

CVPixelBufferLockBaseAddress(depthMapPixelBuffer!, .init(rawValue: 0))
let baseAddress = CVPixelBufferGetBaseAddressOfPlane(depthMapPixelBuffer!, 0)
let data = NSData(bytes: baseAddress, length: size);
... code code code ...

let dic: [AnyHashable: Any] = [kCGImageAuxiliaryDataInfoDataDescription: info,
                                   kCGImageAuxiliaryDataInfoData: data,
                                   kCGImageAuxiliaryDataInfoMetadata: metadata]