ImageMagick .net c# mp4/webm video getting N frame without artifacts?

Question

Trying to make some manipulation with frames using ImageMagick lib on .net (resizing + composing with other image), but faced with incorrect "re-assembling" video, output video coming with some artifacts.

So, as a sample of mp4 short file we have : mp4 (130kb) : https://c2n.me/4eU1Twz.mp4

first frame is : second frame is :

and all next frames with similar artifacts.

Code to get frames and save them to disk :

string incomingFilePath = AppContext.BaseDirectory + "test.mp4";                   

using (var videoFrames = new MagickImageCollection(incomingFilePath, MagickFormat.Mp4))
{                        
        for (var i = 0; i < _incomingImageFrames.Count; i++)
        {
            _incomingImageFrames[i].Write(AppContext.BaseDirectory + Guid.NewGuid() + ".png");
        }
}

In case i am trying to "re-assemble" images to mp4 (with some modifications, using composing() method and etc), i am getting video with artifacts. Sample from output .mp4 :

So need to fix it somehow, but not sure why. Can anyone help me with it?

Answer 1

The reason is in "nature" of mp4/h.264 codec. First frame is always "full" and all others - are the difference between current and previous frame, so as i understand, when mp4 is trying "assemble" not original images (modified) - ffmpeg trying to do the best and glue them all to video.

So as a workaround - we can just remove this frame difference at runtime before forwarding them to ffmpeg engine.

In case of ImageMagick, we need to compose current frame with previous to obtain "full drew frame" as a result before assembling.

Sample code :

string incomingFilePath = AppContext.BaseDirectory + "test.mp4";                   
    
    using (var videoFrames = new MagickImageCollection(incomingFilePath, MagickFormat.Mp4))
    {
            var _resultImageCollection = new MagickImageCollection(); //temp collection
            var prevFullFrame = _incomingImageFrames.First(); //save last full frame, as initial it will be first in collection
            for (var i = 0; i < _incomingImageFrames.Count; i++)
            {
                prevFullFrame.Composite(_incomingImageFrames[i],_incomingImageFrames[i].Page.X, _incomingImageFrames[i].Page.Y, CompositeOperator.Over); //combining previous (full frame) with current (with partial filled pixels)
                _resultImageCollection.add(prevFullFrame); //adding to result collection
            }
            _resultImageCollection.Write(AppContext.BaseDirectory + "result.mp4"); //assembling mp4
    }

Maybe not the best from performance side, but it works fine. Also, it will some increase size, but not dramatically. In my case it was about +10%, so acceptable. (36Kb vs 40Kb)

Update : it will have the same size if you will call Optimize() for collection before save.

_resultImageCollection.Optimize();

Maybe anyone know better solution?