Missing piece between libjpeg-turbo & h264 ffmpeg C/C++

Question

On the left side I have a buffer with decoded pixels that I can get in two formats:

RGB interleaved/packed where bytes in buffer are R0G0B0R1G1B1....

or

YUV444 interleaved/packed where bytes in buffer are Y0U0V0Y1U1V1...

(JCS_RGB or JCS_YCbCr in jpeglib.h)

(Please note that I use libjpeg-turbo because I need to decompress a cropped region of the image. (jpeg_crop_scanline()))

On the right side I have x264 codec via ffmpeg that support only planar pixel formats:

yuv420p, yuvj420p, yuv422p, yuvj422p, yuv444p, yuvj444p, nv12, nv16, nv21, yuv420p10le, yuv422p10le, yuv444p10le, nv20le

yuv444p where bytes in buffer are Y0Y1Y2...U0U1...V0V1...

according to ffmpeg -h encoder=libx264

I have some ideas already:

• Decompress Jpeg to RBG888 in buffer 1 then libswscale to yuv420p in buffer 2 and encoding. (copy)
• Decompress Jpeg to YUV444 interleaved in buffer 1 then SSSE3 magic in buffer 1 to yuv444p and encoding. (no copy)
• or else.

What would be the most effective fastest way ?

I which to avoid buffer copy.

Movie have the same width & height than Jpegs.

Answer 1

If you are fine with using the RGB version of the libx264 encoder, you can use the rgb24 and bgr24 pixel formats without conversion.

ffmpeg -h encoder=libx264rgb
...
    Supported pixel formats: bgr0 bgr24 rgb24

Answer 2

JPEG images are normally stored in YUV420 pixel format.
Selecting JCS_RGB means that libjpeg applies YUV to RGB color conversion.
Assuming the H.264 encoded stream has to be in YUV444 pixel format, we may want to save the time of converting YUV to RGB and back from RGB to YUV.

The answer is going to show how to convert data ordered y,u,v,y,u,v to planar YUV444p format using sws_scale.

The solution uses AV_PIX_FMT_RGB24 to AV_PIX_FMT_GBRP conversion.
Since GBRP (planar GBR) is rarely used, the implementation may not be well optimized.
I don't know if it's going to be faster or slower compared to other solutions...

---

• RGB24 pixel format in ordered r,g,b,r,g,b... and pretends to be y,u,v,y,u,v...
• GBRP pixel format is G plane, B plane, R plane and and pretends to be Y plane, U plane, V plane

The planes order is G,B,R instead of R,G,B.
We may adjust the planes order by setting the orders of the pointers to the planes.

---

Start by creating input sample in y,u,v,y,u,v... format (using FFmpeg CLI):

ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -filter_complex "format=yuvj444p,extractplanes=y+u+v[y][u][v];[u][v][y]mergeplanes=0x001020:gbrp" -f rawvideo -pix_fmt rgb24 yuvyuv.yuv

The file yuvyuv.yuv is used as input for testing the solution (testing without using libjpeg).

---

The following code sample, reads the input, uses sws_scale for converting data ordered y,u,v to planar YUV, and stores the output to binary file yuv_planar.yuv.

Code sample:

#include <stdio.h>
#include <string.h>
#include <stdint.h>

extern "C"
{
#include <libavcodec/avcodec.h>
#include <libswscale/swscale.h>
}


void convertYUV444ToYUV444p(const unsigned char *yuv_in, int width, int height)
{
    struct SwsContext* sws_context = nullptr;
    const int in_linesize[1] = {3 * width}; // YUV ordered stride (as y,u,v,y,u,v...)
    const int out_linesize[3] = {width, width, width}; // YUV planar stride (Y plane, U plane, V plane).

    //Allocate frame for storing YUV planar.
    ////////////////////////////////////////////////////////////////////////////    
    AVFrame* pYUVFrame = av_frame_alloc();

    pYUVFrame->format = AV_PIX_FMT_YUVJ444P; //AV_PIX_FMT_YUV444P;  //Select YUVJ444 (instead of YUV444) because the data is in "JPEG YUV format".
    pYUVFrame->width = width;
    pYUVFrame->height = height;
    int sts = av_frame_get_buffer(pYUVFrame, 0);

    if (sts < 0)
    {
        return;  //Error!
    }
    ////////////////////////////////////////////////////////////////////////////


    //Convert ordered YUV to planar YUV
    //Use RGB24 to GBRP conversion (GBRP is planar format: Green plane, Blue plane, Red plane).
    ////////////////////////////////////////////////////////////////////////////
    sws_context = sws_getContext(width, height,
                                 AV_PIX_FMT_RGB24, width, height,
                                 AV_PIX_FMT_GBRP, 0, nullptr, nullptr, nullptr);

    const uint8_t* in_planes[1] = {yuv_in};

    //Reorder the pointers, to match the GBR planes order (first G [U], then B [V] then R [Y])
    //Y is the applies R channel, U is the applies G and V is the applies B channel of GBRP
    uint8_t* out_planes[3] = {pYUVFrame->data[1], pYUVFrame->data[2], pYUVFrame->data[0]};

    int response = sws_scale(sws_context, in_planes, in_linesize,
                             0, height, out_planes, out_linesize);

    if (response < 0)
    {
        printf("Error: sws_scale response = %d\n", response);
        return;
    }
    ////////////////////////////////////////////////////////////////////////////


    //Write YUV planar output to binary file (for testing)
    ////////////////////////////////////////////////////////////////////////////
    FILE* f = fopen("yuv_planar.yuv", "wb");
    fwrite(pYUVFrame->data[0], 1, width*height, f);
    fwrite(pYUVFrame->data[1], 1, width*height, f);
    fwrite(pYUVFrame->data[2], 1, width*height, f);
    fclose(f);

    //Convert to PNG image for testing (the PNG image is in RGB color space):
    //ffmpeg -y -f rawvideo -video_size 192x108 -pixel_format yuvj444p -i yuv_planar.yuv rgb_image.png
    ////////////////////////////////////////////////////////////////////////////


    sws_freeContext(sws_context);
    av_frame_free(&pYUVFrame);
}



int main()
{  
    //Build input pattern in y,u,v,y,u,v format (for testing):
    //ffmpeg -y -f lavfi -i testsrc=size=192x108:rate=1:duration=1 -filter_complex "format=yuvj444p,extractplanes=y+u+v[y][u][v];[u][v][y]mergeplanes=0x001020:gbrp" -f rawvideo -pix_fmt rgb24 yuvyuv.yuv

    const int width = 192;
    const int height = 108;
    uint8_t *yuv_in = (uint8_t*)av_malloc(width*height*3);

    //Read input image for binary file (for testing)
    ////////////////////////////////////////////////////////////////////////////
    FILE* f = fopen("yuvyuv.yuv", "rb");
    fread(yuv_in, 1, width*height*3, f);
    fclose(f);
    ////////////////////////////////////////////////////////////////////////////


    convertYUV444ToYUV444p(yuv_in, width, height);

    av_free(yuv_in);

    return 0;
}

---

For testing, we may convert the output to PNG image (using FFmpeg CLI):

ffmpeg -y -f rawvideo -video_size 192x108 -pixel_format yuvj444p -i yuv_planar.yuv rgb_image.png

---

Input yuvyuv.yuv as Grayscale PNG image:

Output yuv_planar.yuv as Grayscale PNG image:

rgb_image.png (yuv_planar.yuv converted to colored PNG image):

Answer 3

You can directly decompress a jpeg image into a yuv420p planar buffer by using v_samp_factor and jpeg_read_raw_data(). See gst_jpeg_dec_decode_direct() of GStreamer.