Right way to use vmaf with ffmpeg

Question

I am trying to calculate the VMAF score of a processed video wrt the original file.

Command I have used:

ffmpeg -y -loglevel info -stats -i original.mp4 -i processed.mp4 -lavfi "[0]null[refdeint];[refdeint]scale=1920:1080:flags=neighbor[ref];[1]setpts=PTS+0.0/TB[b];[b]scale=1920:1080:flags=neighbor[c];[c][ref]libvmaf=log_fmt=json:phone_model=1:model_path={model_path_here}/vmaf_v0.6.1.json:n_subsample=1:log_path=log.json" -f null -

Now as per the official documentation of vmaf with ffmpeg found here, it says source/reference file followed by the encoded/distorted/processed file.

But almost all of the blogs I came across, they are using the other way round order of the args, i.e. processed file followed by the original file.

Few examples:

1. https://medium.com/@eyevinntechnology/keep-an-eye-on-the-video-quality-b9bcb58dd5a1: search for "Using VMAF within FFMPEG" in it.

2. https://websites.fraunhofer.de/video-dev/calculating-vmaf-and-psnr-with-ffmpeg/: search for "Metric Calculation with FFmpeg" in it.

EDIT

NOTE: Changing the order does change the VMAF score.

Answer 1

It does not really mater how you are specifying source files in command line. It is important how video streams supplied to libvmaf filter.

By default, source specified first in command line will become stream with index=0, source specified second -- stream with index=1.

You can specify the how these streams are supplied to libvmaf filter, but you do not have to. If you have not done this, the streams with index 0 and 1 will be supplied to the filter in that order.

Lets look at your examples.

Netflix manual (I removed not needed parts):

ffmpeg ... -i src01_hrc00_576x324.yuv \
       ... -i src01_hrc01_576x324.yuv \
    -lavfi "[0:v]setpts=PTS-STARTPTS[reference]; \
            [1:v]setpts=PTS-STARTPTS[distorted]; \
            [distorted][reference]libvmaf=model_path={your_vmaf_dir}/model/vmaf_v0.6.1.json" \
    -f null -

Notice that video stream with index=0 (the one that was specified first in command line) is renamed to reference ([0:v]...[reference]) and the video stream with index=1 (the one that is the second in command line) renamed as distorted ([1:v]...[distorted]).

But when they are sending to libvmaf in the order is: distorted then reference ([distorted][reference]libvmaf).

Article on medium:

ffmpeg -i distorted-file -i reference-file -lavfi libvmaf -f null –

FFMpeg manual:

ffmpeg -i main.mpg -i ref.mpg -lavfi libvmaf -f null -

Distorted file have index=0 Reference file have index=1. As there is no any renaming/remapping, they will be sent to libvmaf in original order: 0 then 1 (distorder then reference). It is equal to [0][1]libvmaf. So video streams order supplied to libvmaf is exactly the same as in Netflix manual.

It is better to keep original order if you compare two videos only as otherwise people might be confused.

However, I do have opposite order in my FFmpeg GUI (FFMetrics). The reason is -- it is possible to specify a multiple distorted files to the program so it is easier to understand and use: ffmetrics.exe [options] ref distorted1 [distorted2] [distorted3] [distorted4] ...

Your command line is also correct: ffmpeg -i reference -i distorted [0]...[refdeint];[refdeint]...[ref];1...[b];[b]...[c];[c][ref]libvmaf...

After all re-mapping streams supplied to libvmaf in correct order: distorted then reference. You must not swap two "-i file" options there as this will change your logic dramatically.

P.S. This was simplified version of what is actually happening :)

Answer 2

The FFmpeg filter expects the first input to be the encoded video stream and the 2nd input to be the reference stream.

It may be (I haven't checked) that the VMAF computation is symmetric with relation to inputs, so in effect, the order may not matter.