Pinch and zoom UIView without transform

Question

I have an UIView placed over an UIImageView and I am trying to pinch-zoom the UIView over the image view without applying transform. The code is as below where something goes wrong which makes the UIview zoom-in/out drastically and moves out of the image view when even a little pinch is applied. What am I doing wrong ?

@objc func didPinch(sender: UIPinchGestureRecognizer) {     
    guard let subView = sender.view else { return }
    var subViewFrame = subView.frame
    let subViewCenter = subView.center
    if sender.state == .changed {
        let scale = sender.scale
        subView.frame = CGRect(x: 0, y: 0, width: subViewFrame.width * scale, height: subViewFrame.height * scale)
        subView.center = subViewCenter
    }
}

Note: I understand that applying transform is a straight forward and an easy way to scale the view but I am not opting that as my UIView has border width applied to it and when the UIView is scaled, the scaling applies to the border width too, which I am looking to avoid.