How to extract frames from video using webcodecs from chrome 86

Question

WebCodecs is released in Chrome 86. But there's no real code example on how to use it yet. Given a video url, how to extract video frames as ImageData using webcodecs?

Answer 1

What you describe is the entire complex process of acquiring raw bitmap-like data (e.g. something you can dump on a canvas), from a formatted file or a stream of data chunks.

In case of files (including the case where your URL points to a complete file, such as an .mp4 file), this is generally made of 2 steps:

1. Parsing the container file into individual chunks of encoded video and/or audio
2. Decoding these chunks of encoded video/audio

WebCodecs only facilitates step 2 of this process, i.e. what is called decoding. The reasoning behind this decision was that parsing the container is computationally trivial, so you can efficiently do this with the File APIs already, but you still need to implement parsing/processing the container yourself.

Luckily, plenty of libraries exist already, many of which ironically existed long before the emergence of the WebCodecs API.

---

MP4Box is one example, helping you acquire encoded video and audio chunks, which you can then feed into a VideoDecoder or AudioDecoder.

With MP4Box, the key piece of your code will be centered around the onSamples callback you provide, and it'll look something like this:

        mp4BoxFile.onSamples = (trackId, user, chunks) =>
        {
            for (let i = 0; i < chunks.length; i++)
            {
                let chunk = chunks[i];
                let encodedChunk = new EncodedVideoChunk({
                    // you'll need to deep-inspect chunk to figure these out
                    type: "key", // or "delta"
                    timestamp: ...
                    duration: ...
                    data: chunk.data
                });

                // pass encodedChunk to a VideoDecoder instance's decode method
            }
        };

This is just a rough sketch of how your code will probably look, it probably won't work without more inspection, and it'll take a lot of trial and error, because this is very low level stuff.

WebCodecs is not the silver bullet you probably expected, but it can help you build one.