Draw cv::Mat onto another cv::Mat at a certain position? (OpenCV C++)

Question

I'm attempting to overlay an image stored in a cv::Mat object onto another cv::Mat image at a certain position using OpenCV's C++ library.

Most of the existing solutions are documented using OpenCV's Python library, and I'm encountering some difficulties translating those solutions to C++.

---

As a minimal test, suppose I have a simple test image test_1.png of size 100x100:

which I want to overlay in the middle of a black background of size 300x300 as such:

Most of the answers, including this one, make use of the addWeighted function. Attempting to implement the function as such:

#include <opencv2/opencv.hpp>
#include <opencv2/core/mat.hpp>
#include <opencv2/core/core.hpp>

int main()
{
    cv::Mat base = cv::Mat::zeros(300, 300, CV_8UC3);
    cv::Mat overlay = cv::imread("test_1.png", 1);
    cv::Mat result;

    cv::addWeighted(base, 1, overlay, 0.1, 0, result);

    cv::imshow("test", result);
    cv::waitKey(1);
    std::system("pause");

    return 0;
}

Results in a C++ memory error at the addWeighted call, I'm assuming because base and overlay aren't the same size.

---

Another post answer that asked about the same issue in C++ specificaly provides the following answer:

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/core/mat.hpp>
#include <opencv2/core/core.hpp>

void OverlayImage(cv::Mat& base, cv::Mat& overlay, std::pair<int, int> pos)
{
    cv::Size fsize{overlay.size()};
    cv::Mat gray;
    cv::Mat mask;
    cv::Mat maskInv;
    cv::cvtColor(overlay, gray, cv::COLOR_BGR2GRAY);
    cv::threshold(gray, mask, 0, 255, cv::THRESH_BINARY);
    cv::bitwise_not(mask, maskInv);

    cv::Mat roi{base(cv::Range(pos.first, pos.second + fsize.width), cv::Range(pos.second, fsize.height))};
    cv::Mat backBg;
    cv::Mat frontBg;
    cv::bitwise_and(roi, roi, backBg, maskInv);
    cv::bitwise_and(overlay, overlay, frontBg, mask);

    cv::Mat result;
    cv::add(backBg, frontBg, result);
    cv::addWeighted(roi, 0.1, result, 0.9, 0.0, result);
    result.copyTo(base(cv::Rect(pos.first, pos.second, fsize.width, fsize.height)));
}

int main()
{
    cv::Mat base = cv::Mat::zeros(300, 300, CV_8UC3);
    cv::Mat overlay = cv::imread("test_1.png", 1);
    
    OverlayImage(base, overlay, std::pair<int, int>(0, 0));

    cv::imshow("test", base);
    cv::waitKey(1);
    std::system("pause");

    return 0;
}

While this solution appears to work with a position of (0, 0):

It doesn't appear to work for any other position other than (0, 0).

If either of the position components are positive, a C++ memory error occurs at the line:

cv::bitwise_and(roi, roi, backBg, maskInv);

If either of the position components are negative, an error occurs within the cv::Mat constructor, from the line:

cv::Mat roi{base(cv::Range(pos.first, pos.second + fsize.width), cv::Range(pos.second, fsize.height))};

---

What is the proper way to overlay a cv::Mat image onto another cv::Mat image at a certain position using OpenCV's C++ library? (sidenote: the background will not always be black as in my example, and I wish for the image to be partially visible if I place it close to the base images' edges)

Thanks for reading my post, any guidance is appreciated.

Answer 1

cv::Mat roi{base(cv::Range(pos.first, pos.second + fsize.width), cv::Range(pos.second, fsize.height))};

This line is a mess. Maybe, should be

cv::Mat roi{base(cv::Range(pos.second, pos.second + fsize.height), cv::Range(pos.first, pos.first+fsize.width))};

---

If you want to support the situation that overlay is not completely included in base, like this:

inline cv::Range OverlapRange( const cv::Range &A, const cv::Range &B )
{   return cv::Range( std::max( A.start, B.start ), std::min( A.end, B.end ) ); }

inline bool IsEmpty( const cv::Range &R ){  return ( R.end <= R.start );    }

void OverlayImage( cv::Mat& base, cv::Mat& overlay, std::pair<int, int> pos )
{
    auto BaseRowRange = OverlapRange( cv::Range{ 0, base.rows }, { pos.second, pos.second+overlay.rows } );
    auto BaseColRange = OverlapRange( cv::Range{ 0, base.cols }, { pos.first, pos.first+overlay.cols } );
    if( IsEmpty( BaseRowRange ) || IsEmpty( BaseColRange ) )return;

    auto Base_roi = base(BaseRowRange,BaseColRange);
    auto OV_roi = overlay(
        cv::Rect(
            BaseColRange.start - pos.first,
            BaseRowRange.start - pos.second,
            BaseColRange.end - BaseColRange.start,
            BaseRowRange.end - BaseRowRange.start
        )
    );

    cv::Mat BkgndMask;
    cv::inRange( OV_roi, cv::Scalar(0,0,0), cv::Scalar(0,0,0), BkgndMask );

    auto Result = Base_roi.clone();
    OV_roi.copyTo( Result, ~BkgndMask );

    cv::addWeighted( Base_roi, 0.1, Result, 0.9, 0, Base_roi );
}

For example, when main() is below, the result becomes:

int main()
{
    //base is small green image
    cv::Mat base = cv::Mat(60, 60, CV_8UC3);
    base = cv::Scalar( 0,128,0 );
    //This is the your test image
    cv::Mat overlay = cv::imread("test1.png", 1);

    OverlayImage( base, overlay, std::pair<int, int>(-30, 25) );
    cv::imshow("test", base);
    cv::waitKey(0);
    return 0;
}

Answer 2

Your slicing using cv::Range is not right - you are slicing a smaller image than the "overlay". In fact, you don't need the slicing at all. You can read the image directly as BGRA, use the alpha channel to create a binary mask via thresholding, then, AND the mask with the original BGR channels.

Next, use the "base" image as canvas and paste the masked image anywhere inside it. Be careful to paste it within the canvas dimensions, otherwise you'll get an exception. Use the copyTo method from the cv::Mat class.

Here's your code modified:

void OverlayImage(cv::Mat& base, cv::Mat& overlay, std::pair<int, int> pos)
{
    // Create overlay mask:
    cv::Mat channels[4];
    // Split channels:
    cv::split(overlay, channels);

    // Threshold alpha:
    cv::Mat mask;
    cv::threshold(channels[3], mask, 0, 255, cv::THRESH_OTSU);

    // Create BGR overlay:
    cv::Mat BGR;
    std::vector<cv::Mat> temp = {channels[0], channels[1], channels[2]};
    cv::merge(temp, BGR);

    // And mask an overlay:
    cv::bitwise_and(BGR, BGR, mask);

    // Get "overlay" dimensions:
    int overlayWidth = BGR.cols;
    int overlayHeight = BGR.rows;

    // Paste "overlay" in canvas at position pos:
    // Hint: small_image.copyTo(big_image(cv::Rect(x,y,small_image.cols, small_image.rows)));
    BGR.copyTo(base(cv::Rect(pos.first, pos.second, overlayWidth, overlayHeight)));
}

int main()
{

    std::string imagePath = "D://opencvImages//ocpaKm.png";
    cv::Mat base = cv::Mat::zeros(300, 300, CV_8UC3);
    cv::Mat overlay = cv::imread(imagePath, cv::IMREAD_UNCHANGED);

    // Paste at center of 300 x 300 image:
    int x = 0.5 * (base.cols - overlay.cols);
    int y = 0.5 * (base.rows - overlay.rows);

    // Get final image:
    OverlayImage(base, overlay, std::pair<int, int>(x, y));

    cv::imshow("Out Image", base);
    cv::waitKey(0);


    return 0;
}

This is the result trying to paste the image at the center of the canvas:

Again, be careful while using copyTo because if, somehow, the smaller image gets pasted outside the larger image (even one row/col) you'll raise an exception, so a formal check for these conditions is advised before you attempt to paste the image.