first memcpy into buffer faster than direct access to pixels of CVPixelBufferGetBaseAddress

Question

I noticed that accessing the pixels returned by CVPixelBufferGetBaseAddress directly (I'm using two nested for-loops) is about 100 times slower than first allocating a buffer with malloc, memcpy the pixels into that buffer and then accessing the pixels in that buffer. Why is that?