What is the best way to process video and transmitt on imx6 board?

Question

Recently, I'm trying to use a board with imx6 processor and a MIPI-CSI camera to transmit video.

Our goal is to transmit the video (1920x1080@30fps) through the RTP protocol and we did it using gstreamer pipelines. The command pipeline to transmit is:

gst-launch-1.0 imxv4l2videosrc device=/dev/video1 imx-capture-mode=2 ! imxvpuenc_h264 quant-param=29 ! rtph264pay mtu=60000 ! udpsink host=10.42.0.1 port=5000 sync=false

The command transmit the video with a resolution of 1920x108 (imx-capture-mode=2 parameter).

And from the PC (10.42.0.1) we catch the video (trough port 5000) using the command:

gst-launch-1.0 -v udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 !  rtph264depay ! avdec_h264 ! fpsdisplaysink text-overlay=false sync=false &

And the flow of video is very well.

But now, we want process the video before it is transmitted, because the camera is in movement and we need to add some algorithms to reduce the noise due to the shaking.

In other words, the main idea is: take the video from camera->process the video-> transmit to RTP. All of it using the board with the imx6.

The board is based on the linux yocto project, and it have opencv 3.1 with gstreamer plugin enabled. Our idea is to take the video using opencv and gstreamer pipelines, process and transmit it... But first we did some test taking the video and transmiting it without process it, but when we catch the video from the PC, it is very slow and delayed...

We want to know if that is the right way for our goal. Here is the source code that we are using:

#include <QCoreApplication>
#include <QDebug>
#include <opencv2/opencv.hpp>
#include <opencv2/video.hpp>

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    cv::VideoCapture cap("imxv4l2videosrc device=/dev/video1 imx-capture-mode=2 ! videoconvert ! appsink");
    Q_ASSERT(cap.isOpened());
    if(!cap.isOpened())
    {
        qDebug() << "Error with the video capturer";
        a.exit(-1);
    }
    cv::Mat frame;
    cap >> frame; //Dummy read
    double frame_width=   cap.get(CV_CAP_PROP_FRAME_WIDTH);
    double frame_height=   cap.get(CV_CAP_PROP_FRAME_HEIGHT);

    cv::VideoWriter writer;
    writer.open("appsrc ! videoconvert ! imxvpuenc_h264 quant-param=29 ! rtph264pay mtu=60000 ! udpsink host=10.42.0.1 port=5000 sync=false",
                0,30,cv::Size(frame_width,frame_height));
    if(!writer.isOpened())
    {
        qDebug() << "Error video writer";
        a.exit(-1);
    }

    while(true) {

            cap.read(frame);

            if(frame.empty())
                break;

            writer.write(frame);
            cv::waitKey(33);
        }

    return a.exec();
}

Answer 1

I am guessing the reason your video is slowing down is video format conversions. OpenCv acquires BGR formatted video from gstreamer appsink element. If your imxv4l2videosrc outputs a different format (for example UYVY or I420), videoconvert element will have to do a lot of calculation. In some cases these calculations use more than 100% of your CPU, and this will slow down your input stream.

You can check what format your original pipe is using by adding -v to the end of your pipe.

gst-launch-1.0 imxv4l2videosrc device=/dev/video1 imx-capture-mode=2 ! imxvpuenc_h264 quant-param=29 ! rtph264pay mtu=60000 ! udpsink host=10.42.0.1 port=5000 sync=false -v

We can test if video conversion is slowing down your system by reducing the resoulution. Can you try again by lowering the resoulution to 1280x720, or 640x360, and check your CPU usage?

If this is the case we can request output from imxv4l2videosrc with different formats. I havent used that element, and I don't know its capabilities. But you can check the capabilities with gst-inspect-1.0 command.

gst-inspect-1.0 imxv4l2videosrc

The output will have a part like this

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:

      video/x-raw
                 format: { (string)RGB16, (string)BGR, (string)RGB, (string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)YVU9, (string)YV12, (string)YUY2, (string)YVYU, (string)UYVY, (string)Y42B, (string)Y41B, (string)YUV9, (string)NV12_64Z32, (string)NV24, (string)NV61, (string)NV16, (string)NV21, (string)NV12, (string)I420, (string)BGRA, (string)BGRx, (string)ARGB, (string)xRGB, (string)BGR15, (string)RGB15 }
                  width: [ 1, 32768 ]
                 height: [ 1, 32768 ]
              framerate: [ 0/1, 2147483647/1 ]

Formats like BGRx, 'RGB', or 'xRGB' that have B,G,and R channels will be useful and fast for you.