Cropping AVCapturePhoto to overlay rectangle displayed on screen

Question

I am trying to take a picture of a thin piece of metal, cropped to the outline displayed on the screen. I have seen almost every other post on here, but nothing has got it for me yet. This image will then be used for analysis by a library. I can get some cropping to happen, but never to the rectangle displayed. I have tried rotating the image before cropping, and calculating the rect based on the rectangle on screen.

Here is my capture code. PreviewView is the container, videoLayer is for the AVCapture video.

    // Photo capture delegate
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        guard let imgData = photo.fileDataRepresentation(), let uiImg = UIImage(data: imgData), let cgImg = uiImg.cgImage else {
            return
        }
        print("Original image size: ", uiImg.size, "\nCGHeight: ", cgImg.height, " width: ", cgImg.width)
        print("Orientation: ", uiImg.imageOrientation.rawValue)
        
        guard let img = cropImage(image: uiImg) else {
            return
        }
        
        showImage(image: img)
    }

    
    func cropImage(image: UIImage) -> UIImage? {
        print("Image size before crop: ", image.size)
        //Get the croppedRect from function below
        let croppedRect = calculateRect(image: image)
        guard let imgRet = image.cgImage?.cropping(to: croppedRect) else {
            return nil
        }
        return UIImage(cgImage: imgRet)
    }
    
    func calculateRect(image: UIImage) -> CGRect {
        let originalSize: CGSize
        let visibleLayerFrame = self.rectangleView.bounds

        // Calculate the rect from the rectangleview to translate to the image
        let metaRect = (self.videoLayer.metadataOutputRectConverted(fromLayerRect: visibleLayerFrame))
        print("MetaRect: ", metaRect)
        // check orientation
        if (image.imageOrientation == UIImage.Orientation.left || image.imageOrientation == UIImage.Orientation.right) {
            originalSize = CGSize(width: image.size.height, height: image.size.width)
        } else {
            originalSize = image.size
        }

        let cropRect: CGRect = CGRect(x: metaRect.origin.x * originalSize.width, y: metaRect.origin.y * originalSize.height, width: metaRect.size.width * originalSize.width, height: metaRect.size.height * originalSize.height).integral
        print("Calculated Rect: ", cropRect)
        return cropRect
    }

func showImage(image: UIImage) {
    if takenImage != nil {
        takenImage = nil
    }
    takenImage = UIImageView(image: image)
    takenImage.frame = CGRect(x: 10, y: 50, width: 400, height: 1080)
    takenImage.contentMode = .scaleAspectFit
    print("Cropped Image Size: ", image.size)
    self.previewView.addSubview(takenImage)
}

And here is along the line of what I keep getting.

What am I screwing up?

Answer 1

I managed to solve the issue for my use case.

private func cropToPreviewLayer(from originalImage: UIImage, toSizeOf rect: CGRect) -> UIImage? {
    guard let cgImage = originalImage.cgImage else { return nil }

    // This previewLayer is the AVCaptureVideoPreviewLayer which the resizeAspectFill and videoOrientation portrait has been set.
    let outputRect = previewLayer.metadataOutputRectConverted(fromLayerRect: rect)

    let width = CGFloat(cgImage.width)
    let height = CGFloat(cgImage.height)

    let cropRect = CGRect(x: (outputRect.origin.x * width), y: (outputRect.origin.y * height), width: (outputRect.size.width * width), height: (outputRect.size.height * height))

    if let croppedCGImage = cgImage.cropping(to: cropRect) {
        return UIImage(cgImage: croppedCGImage, scale: 1.0, orientation: originalImage.imageOrientation)
    }

    return nil
}

usage of the piece of code for my case:

let rect = CGRect(x: 25, y: 150, width: 325, height: 230)
let croppedImage = self.cropToPreviewLayer(from: image, toSizeOf: rect)
self.imageView.image = croppedImage

Answer 2

The world of UIKit has the TOP LEFT corner as 0,0.

The 0,0 point in the AVFoundation world is the BOTTOM LEFT corner.

So you have to translate by rotating 90 degrees.

That's why your image is bonkers.

Also remember that because of the origin translation the following rules apply:

• X is actually up and down
• Y is actually left and right
• width and height are swapped

Also be aware that the UIImageView content mode setting WILL impact how your image scales. You might want to use .scaleAspectFill and NOT AspectFit if you really want to see how your image looks in the UIView.

I used this code snippet to see what was behind the curtain:

// figure out how to cut/crop this
let realImageRect = AVMakeRect(aspectRatio: image.size, insideRect: (self.cameraPreview?.frame)!)
NSLog("real image rectangle = \(realImageRect.debugDescription)")

The 'cameraPreview' reference above is the control you're using for your AV Capture Session.

Good luck!