How to stream frames from OpenCV C++ code to Video4Linux or ffmpeg?

Question

I am experimenting with OpenCV to process frames from a video stream. The goal is to fetch a frame from a stream, process it and then put the processed frame to a new / fresh stream.

I have been able to successfully read streams using OpenCV video capture functionality. But do not know how I can create an output stream with the processed frames.

In order to do some basic tests, I created a stream from a local video file using ffmpeg like so:

ffmpeg -i sample.mp4 -v 0 -vcodec mpeg4 -f mpegts \
        "udp://@127.0.0.1:23000?overrun_nonfatal=1&fifo_size=50000000"

And in my C++ code using the VideoCapture functionality of the OpenCV library, I am able to capture the above created stream. A basic layout of what I am trying to achieve is attached below:

cv::VideoCapture capture("udp://@127.0.0.1:23000?overrun_nonfatal=1&fifo_size=50000000", cv::CAP_FFMPEG);

cv::Mat frame;

while (true) 
{
    // use the above stream to capture a frame
    capture >> frame;
    
    // process the frame (not relevant here)
    ...

    // finally, after all the processing is done I 
    // want to put this frame on a new stream say at
    // udp://@127.0.0.1:25000, I don't know how to do
    // this, ideally would like to use Video4Linux but
    // solutions with ffmpeg are appreciated as well
}

As you can see from comment in above code, I don't have any idea how I should even begin handling this, I tried searching for similar questions but all I could find was how to do VideoCapture using streams, nothing related to outputting to a stream.

I am relatively new to this and this might seem like a very basic question to many of you, please do excuse me.

Answer 1

We may use the same technique as in my following Python code sample.

• Execute FFmpeg as sub-process, open stdin pipe for writing

   FILE *pipeout = popen(ffmpeg_cmd.c_str(), "w")
  

• Write frame.data to stdin pipe of FFmpeg sub-process (in a loop)

   fwrite(frame.data, 1, width*height*3, pipeout);
  

• Close the pipe at the end (it is going to close the sub-process)

   pclose(pipeout);
  

---

The following sample is a generic example - building numbered frames, and writing the encoded video to an MKV output file.

The example uses the following equivalent command line:
ffmpeg -y -f rawvideo -r 10 -video_size 320x240 -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 24 -pix_fmt yuv420p output.mkv

You may adjust the arguments for your specific requirements (replace output.mkv with udp://@127.0.0.1:25000).

Replace the numbered frame with capture >> frame, and adjust the size and framerate.

---

Code sample:

#include <stdio.h>
#include <chrono>
#include <thread>
#include "opencv2/opencv.hpp"

int main()
{
    int width = 320;
    int height = 240;
    int n_frames = 100;
    int fps = 10;

    //Use a "generic" example (write the output video in output.mkv video file).
    //ffmpeg -y -f rawvideo -r 10 -video_size 320x240 -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 24 -pix_fmt yuv420p output.mkv
    std::string ffmpeg_cmd = std::string("ffmpeg -y -f rawvideo -r ") + std::to_string(fps) +
                             " -video_size " + std::to_string(width) + "x" + std::to_string(height) +
                             " -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 24 -pix_fmt yuv420p output.mkv";

    //Execute FFmpeg as sub-process, open stdin pipe (of FFmpeg sub-process) for writing.
    //In Windows we need to use _popen and in Linux popen
#ifdef _MSC_VER
    FILE *pipeout = _popen(ffmpeg_cmd.c_str(), "wb");   //Windows (ffmpeg.exe must be in the execution path)
#else
    //https://batchloaf.wordpress.com/2017/02/12/a-simple-way-to-read-and-write-audio-and-video-files-in-c-using-ffmpeg-part-2-video/
    FILE *pipeout = popen(ffmpeg_cmd.c_str(), "w");     //Linux (assume ffmpeg exist in /usr/bin/ffmpeg (and in path).
#endif

    for (int i = 0; i < n_frames; i++)
    {
        cv::Mat frame = cv::Mat(height, width, CV_8UC3);
        frame = cv::Scalar(60, 60, 60); //Fill background with dark gray
        cv::putText(frame, std::to_string(i+1), cv::Point(width/2-50*(int)(std::to_string(i+1).length()), height/2+50), cv::FONT_HERSHEY_DUPLEX, 5, cv::Scalar(30, 255, 30), 10);  // Draw a green number

        cv::imshow("frame", frame);cv::waitKey(1); //Show the frame for testing

        //Write width*height*3 bytes to stdin pipe of FFmpeg sub-process (assume frame data is continuous in the RAM).
        fwrite(frame.data, 1, width*height*3, pipeout);
    }

    // Flush and close input and output pipes
    fflush(pipeout);

#ifdef _MSC_VER
    _pclose(pipeout);   //Windows
#else
    pclose(pipeout);    //Linux
#endif

    //It looks like we need to wait one more second at the end. //https://stackoverflow.com/a/62804585/4926757
    std::this_thread::sleep_for(std::chrono::milliseconds(1000)); // sleep for 1 second

    return 0;
}