AVCaptureSession w/o Decompression

Question

Environment

• OS-X
• Yosemite
• Objective-C
• Compressed video input ( AVCaptureDeviceFormat == 'muxx'/'isr ' )

Use-case

Compressed video stream is exposed to the OS as a HAL capture device ( AVCaptureDevice ) supporting the 'muxx/isr ' payload format, I would like to be able to directly access the raw bytes of the compressed payload w/o decompressing.

Implementation

1. Find the device using "+ (AVCaptureDevice *)deviceWithUniqueID:(NSString *)deviceUniqueID"
2. Create 'AVCaptureDeviceInput'
3. Create 'AVCaptureVideoDataOutput'
4. Associate the 'AVCaptureDevice' with 'AVCaptureDeviceInput'
5. setSampleBufferDelegate on 'AVCaptureVideoDataOutput'
6. Create AVCaptureSession
7. Configure the session by adding inputs and outputs ( no preset is selected )
8. start the session by calling [session startRunning]
9. Upon 'AVCaptureVideoDataOutputSampleBufferDelegate' invocation call 'CMSampleBufferGetDataBuffer(sampleBuffer);' to get a reference to the RAW buffer

Problem description

While the delegate is invoked by the system the buffers I get refer to a decoded yuy2 image rather than to the compressed buffer ( before it was de-muxed and de-coded ), and thus, 'CMSampleBufferGetDataBuffer' returns a nil pointer.

How can I get the RAW payload sent by the AVCaptureDevice w/o decompressing?

Answer 1

The solution is to use the C++ CoreMediaIO framework, a sample is provided at this link, the main problem with CoreMediaIO is that it is not widely used and has minimal documentation, the sample code provided by Apple is old and req quite much tinkering before it can compile with recent SDKs, however, in the end, it works, this Blog entry has a nice example.