iOS fast image difference comparison

Question

Im looking for a fast way to compare two frames of video, and decide if a lot has changed between them. This will be used to decide if I should send a request to image recognition service over REST, so I don't want to keep sending them, until there might be some different results. Something similar is doing Vuforia SDK. Im starting with a Framebuffer from ARKit, and I have it scaled to 640:480 and converted to RGB888 vBuffer_image. It could compare just few points, but it needs to find out if difference is significant nicely. I started by calculating difference between few points using vDSP functions, but this has a disadvantage - if I move camera even very slightly to left/right, then the same points have different portions of image, and the calculated difference is high, even if nothing really changed much. I was thinking about using histograms, but I didn't test this approach yet. What would be the best solution for this? It needs to be fast, it can compare just smaller version of image, etc.

I have tested another approach using VNFeaturePointObservation from Vision. This works a lot better, but Im afraid it might be more CPU demanding. I need to test this on some older devices. Anyway, this is a part of code that works nicely. If someone could suggest some better approach to test, please let know:

    private var lastScanningImageFingerprint: VNFeaturePrintObservation?
    
    // Returns true if these are different enough
    private func compareScanningImages(current: VNFeaturePrintObservation, last: VNFeaturePrintObservation?) -> Bool {
        guard let last = last else { return true }
        var distance = Float(0)
        try! last.computeDistance(&distance, to: current)
        print(distance)
        return distance > 10
    }
    
    // After scanning is done, subclass should prepare suggestedTargets array.
    private func performScanningIfNeeded(_ sender: Timer) {
        guard !scanningInProgress else { return } // Wait for previous scanning to finish
        guard let vImageBuffer = deletate?.currentFrameScalledImage else { return }
        guard let image = CGImage.create(from: vImageBuffer) else { return }
        
        func featureprintObservationForImage(image: CGImage) -> VNFeaturePrintObservation? {
            let requestHandler = VNImageRequestHandler(cgImage: image, options: [:])
            let request = VNGenerateImageFeaturePrintRequest()
            do {
                try requestHandler.perform([request])
                return request.results?.first as? VNFeaturePrintObservation
            } catch {
                print("Vision error: \(error)")
                return nil
            }
        }
        
        guard let imageFingerprint = featureprintObservationForImage(image: image) else { return }
        guard compareScanningImages(current: imageFingerprint, last: lastScanningImageFingerprint) else { return }
        print("SCANN \(Date())")
        lastScanningImageFingerprint = featureprintObservationForImage(image: image)
        executeScanning(on: image) { [weak self] in
            self?.scanningInProgress = false
        }
    }

Tested on older iPhone - as expected this causes some frame drops on camera preview. So I need a faster algorithm