ffmpeg memory leak when opening libx264 encoder

Question

I have spotted a memory leak issue when I use the libx264 encoder in the FFmpeg C API. Specifically, when it comes to deallocate memory after encoding a video. After tracking the factor that causes it, I realized that it happens after invoking avcodec_open2, which allocates some memory that afterwards cannot be freed. Once the video is processed, calling avcodec_close and then avcodec_free_context does not entirely free all the allocated memory.

After some investigation, I found out that the problem could be located in AVCodecContext::priv_data being allocated but not being freed afterwards. In this question a solution to the issue is proposed. However, I tried to implement it without success (the memory being leaked seems to be exactly the same).

As a matter of fact, the following simple code (which includes the patch that was proposed in the aforementioned question), in which the codec is being opened and closed multiple times without even writing a single frame or allocating an AVFormatContext, illustrates the memory leak.

#include <stdint.h>
extern "C"{
#include <x264.h>
#include <libavcodec/avcodec.h>
#include <libavutil/opt.h>
}

int main()
{
    avcodec_register_all();

    AVCodec *codec;
    AVCodecContext *c;
    for (int n=0; n<2000; n++)
    {
        codec = avcodec_find_encoder_by_name("libx264");
        c=avcodec_alloc_context3(codec);
        c->pix_fmt=AV_PIX_FMT_YUV420P;
        c->width=1920;
        c->height=1080;
        c->time_base=(AVRational){1, 30};
        c->framerate=(AVRational){30, 1};
        avcodec_open2(c, codec, NULL);
        avcodec_close(c);
        av_opt_free(c->priv_data);
        av_freep(&c->priv_data);
        avcodec_free_context(&c);
    }
    return 0;
}

It must be remarked that if the line codec = avcodec_find_encoder_by_name("libx264") is replaced to an invocation to an internal/native encoder, e.g., codec = avcodec_find_encoder(AV_CODEC_ID_MPEG4), then the memory leak issue completely disappears. Hence, it certainly seems to be an issue related to some private data of the external encoder not being properly freed.

It is also worth mentioning that I am using an old version of ffmpeg and libx264. To be more precise, ffmpeg version 2.8git and libx264 version 0.136.x. For technical reasons that are beyond the scope of this question, it is not possible to upgrade the libraries to newer versions onto the project in which these are being used. I am fully aware that most of the involved ffmpeg/libx264 code has been probably changed along the years and many functions became deprecated or fixed, and thus reporting this as a possible bug in the ffmpeg developer's mailbox is out of the question.

Nevertheless, I am still asking this here because I would like to know whether it is just some mistake on my end and/or something I am not taking into account when it comes to free all the memory relative to an external encoder (best case scenario). Otherwise, I would like to know whether there can be some reasonably cheap solution through some custom code or function that can be implemented as a patch (assuming it is indeed an issue related to ffmpeg/libx264), no matter if it makes the whole deallocation code less elegant or concise. If someone is still working on these older versions of ffmpeg and can come up with a workaround, that would be highly appreciated.

Answer 1

I have been able to spot more accurately the source of the issue, which was completely unrelated to how FFmpeg was handling the encoder data allocation, so x264 was directly the one to blame. Specifically, the call x264_encoder_open was the factor that caused the memory leak, whose magnitude was essentially the same as the one exposed in the question after running the following code:

#include <stdint.h>
extern "C"{
#include <x264.h>
}

int main()
{
    x264_param_t param;
    x264_t *h;

    for (int n=0; n<2000; n++)
    {
        x264_param_default_preset( &param, "medium", NULL );
        param.i_csp = X264_CSP_I420;
        param.i_width  = 1920;
        param.i_height = 1080;
        param.b_vfr_input = 0;
        param.b_repeat_headers = 1;
        param.b_annexb = 1;
        h = x264_encoder_open( &param );
        x264_encoder_close(h);
    }
    return 0;
}

It turned out that until now I didn't realize that the x264 version that I was using (from mid 2013) was significantly older than the version of FFmpeg (from early 2016). Therefore, I gave it a try and upgraded the libx264 sources to a version from early 2016 as well. Fortunately, both versions got along well in terms of compilation and, even better, the memory leak seems to be gone.