What is the fastest way to extract all frames of video into images in C#?

Question

I have created a application to extract every frame from a video. It is not perfect yet and has a lot of problems with it. One of the problems is how long it takes. It took a full 2 hours to extract the frames of a 3 minutes video and the original video was four minutes but it gave me a memory error after the video was nearly finished:

'OpenCV: Failed to allocate 2764800 bytes'

so, it takes a long time to complete the task and it has a few bugs I am using Emgu.CV and MediaToolkit

Here is my code:

  using Emgu.CV;
  using MediaToolkit;
  using MediaToolkit.Model;



    string file;

    private void CreateFrames()
    {
        int i = 0;

        VideoCapture _video = new VideoCapture(file);
        var videoInfo = new MediaFile { Filename = file };
        using (var engine = new Engine()) { engine.GetMetadata(videoInfo); }

        while (i <= videoInfo.Metadata.Duration.Milliseconds * videoInfo.Metadata.VideoData.Fps)
        {
            Mat m = new Mat();
            _video.Read(m);
            _video.SetCaptureProperty(Emgu.CV.CvEnum.CapProp.PosFrames, i);
            preview.Image = _video.QueryFrame().ToBitmap();
            Bitmap bitmap = _video.QueryFrame().ToBitmap();
            bitmap.Save(String.Format(@"C:\\frames\\videoframe-{0}.png", i), 
    ImageFormat.Png);
            i++;
        }
    }

    private async void browse_Click(object sender, EventArgs e)
    {
        DialogResult result = openFileDialog.ShowDialog();
        if (result == DialogResult.OK)
        {
            file = openFileDialog.FileName;
            var videoInfo = new MediaFile { Filename = file };
            using (var engine = new Engine()) { engine.GetMetadata(videoInfo); }
            label1.Text = videoInfo.Metadata.VideoData.Fps + "  \n" + 
         videoInfo.Metadata.Duration.ToString();

            // must be run on a second thread else it will not work and it will freeze
            Task getframes = new Task(CreateFrames);
            getframes.Start();
        }
    }

I would appreciate if somebody has a solution to make this faster and possibly a fix to the memory problem.

Answer 1

First of all, check for IDisposable objects without the "using". For example, Bitmap inherits from Image, that implements IDisposable. So, when using the bimap, your code should be like this:

using (Bitmap bitmap = _video.QueryFrame().ToBitmap())
{
    bitmap.Save(String.Format(@"C:\\frames\\videoframe-{0}.png", i), ImageFormat.Png);
}

Also check if the Mat class implements IDisposable. If it is the case, another using could help you with the memory problem.

For the performance issue, I really cant help, but 3 minutes video, at 60fps, are 10800 frames. 2 hours are 7200 seconds. So 0.6 seconds per frame. Is this really slow? Just asking, I have no idea...

Answer 2

Here is an implementation using just Emgu.CV (I am not familiar with MediaToolKit).

using (var video = new VideoCapture(file))
using (var img = new Mat())
{
    while (video.Grab())
    {
        video.Retrieve(img);
        var filename = Path.Combine(outputDirectory, $"{i}.png");
        CvInvoke.Imwrite(filename, img);
        i++;
    }
}

I tested this on a FullHD mp4 video. Don't know the exact codec. This takes roughly 65ms per frame with my CPU (i7-8700k). But consider if you need to use the png format as it is slow to encode. If you don't want any compression loss and your harddisk is fast and big enough for bmp, it takes just 13ms per frame. jpg takes 40 ms.