Detection I Frame in h264 stream (fragmented mp4)

Question

I need to check the first frame in H264 stream is I-Frame.

On the input I have fragmented mp4 file. I tried to find type of frame in the moof->traf->trun "Sample depends on" flag. But seems, not every container has this flag filled in. So I want to try determine type of frame in the mdat block with raw data.

I need to check only the first frame is I-frame in the each fragment. Information about other frames doesn't matter.

How I can do it?

Answer 1

You can check the NAL unit type. NAL unit type 5 indicates an IDR frame which is an I frame. Inside the 'mdat' the video is stored:

<size><NAL><size><NAL>...<size><NAL>

The lower 5 bits of the first byte of each NAL unit indicates the type. Skip through types 6,7,8 and 9 until you find type 1 (non IDR frame) or type 5 (IDR frame).

MP4 files should not contain start codes ([00] 00 00 01) or access unit delimiters.

MPEG-2 Transport Streams or *.h264 raw contain start codes ([00] 00 00 01) and access code delimiters.

The size field in MP4 is most of the time 4 bytes but if you want the correct answer you have parse the codec private data (SPS/PPS).

In short H.264 comes in two formats:

• Annex-B or
• MP4 (mdat)

Annex-B (MPEG-2 TS, or *.264 raw file):

<[00] 00 00 01> <NAL> <[00] 00 00 01> <NAL> ... <[00] 00 00 01> <NAL>

MP4 (mdat):

<size><NAL><size><NAL>...<size><NAL> 

Your file in https://drive.google.com/file/d/1Vwcz8WsTuRLJie8SFzGspizyTc-caGjc/view?usp=sharing has video and audio in the same mdat.

So to get the I-frame detection reliable you have to parse a little more:

this gives you the video start into mdat:

moof[i]->traf[0]->trun[0]->dataOffset

audio starts here => stop parsing video

moof[i]->traf[1]->trun[0]->dataOffset