Decode MP4 video with VideoDecoder

Question

I want to make a web app, which extracts each frame of a MP4 file, and saves it as a PNG. Setting the currentTime of a <video> element and drawing it into a canvas does not work, as it often skips some frames.

I am using the VideoDecoder. It has a decode method, which receives an EncodedVideoChunk, that contains an array of bytes - a "chunk of a video".

I parsed this MP4 file to get "moov" - "trak" - 0 - "mdia" - "minf" - "stbl". I slice a piece of the MP4 file from stbl.stco[0], of a length stbl.stsz[0] (80 and 42548 in this case). The bytes are

[0, 0, 0, 2,   9, 16, 0, 0,   166, 42, 37, 136,   128, 67, 255, 235, ...]

I still see an error message:

Failed to execute 'decode' on 'VideoDecoder': A key frame is required after configure() or flush().

What am I doing wrong? How can I find a right piece of data, that should be given to the VideoDecoder?

Answer 1

All my steps were correct. The problem was, that the VideoDecoder missed the description parameter during configure(). Seems like no MP4 file can be decoded without it.

Use the sequence of 40 bytes at stbl.stsd.extraData.avcC as a value of the description.

Answer 2

The key frame error refers to the EncodedVideoChunk.type field, which must be "key" at the start of decoding. It's assumed that this information comes from the media container.

This is independent of the format of the bitstream; if the bitstream is Annex B then you should not set a description, and if the stream is AVC (used in MP4), you should set description.