DispatchSemaphore + DispatchQueue not working as expected?

Question

I'm trying to improve the time it takes for a task to finish by leveraging multithreading/paralleling.
I'm reading CMSampleBuffers from a video track, manipulating them, then storing them back to an array for later use. Because each manipulation is "costly" in terms of RAM and CPU, I'm using DispatchSemaphore to limit the number of parallel tasks (5 in the example).

Basically, I'm trying to make the "system" process more than a single frame in a time, and not above 5 so the device won't crash due to memory issues. In the current implementation below it's for some reason doing it almost serialize and not in parallel.
Any help will be highly appreciated!

I tried taking reference from here: How to limit gcd queue buffer size for the implementation.

Code:

class MyService {

  let semaphore = DispatchSemaphore(value: 5)
  let processQueue = DispatchQueue(label: "custom.process", attributes: .concurrent)

    func startReading() {

        for sampleBuffer in sampleBuffers {
            // signal wait
            semaphore.wait()

            // async queue
            processQueue.async {

                // run taks
                self.process(buffer: sampleBuffer) { pixelBuffer in

                    // singal to semaphore
                    self.semaphore.signal()

                }
            }
        }
    }


  func process(buffer: CMSampleBuffer, completion: @escaping (CVPixelBuffer) -> (Void)) {

        // run on a background thread to avoid UI freeze
        DispatchQueue.global(qos: .userInteractive).async {

            // Do something
            // Do something
            // Do something
            // Do something
            completion(processedBuffer)
        }   
       
    }

}