C# green screen removal (chroma key technique) and combining background image/video with webcam stream

166 Views Asked by At

I'm working on an app where I need to overlay a webcam video of people on a green/blue background screen (shades of green/blue may differ) with a still background image or video.

Unfortunately, I can't find any professional solution (in the form of C# framework/library) to solve chroma key image filtering issue.

I decided to write my own solution using the OpenCV library. Using @Burak's solution for Python, I partially solved the issue by porting the code to C#:

private void Worker_DoWork(object sender, DoWorkEventArgs e)
{
    Mat mask, result = null;
    var worker = (BackgroundWorker)sender;
    while (!worker.CancellationPending)
    {
        using (var frame = videoCap.RetrieveMat())
        {
            // Get a chroma key mask
            mask = ChromaKeyMask(frame);

            if (videoBack != null)
            {
                if (videoBack.Width != frame.Width && videoBack.Height != frame.Height)
                {
                    videoBack = videoBack.Resize(frame.Size());
                }

                result = videoBack.Clone();

                frame.CvtColor(ColorConversionCodes.RGB2BGR).CopyTo(result, mask);
            }

            Dispatcher.Invoke(() =>
            {
                FrameImage.Source = frame.ToWriteableBitmap();
                MaskImage.Source = mask.ToWriteableBitmap();
                ResultImage.Source = result?.CvtColor(ColorConversionCodes.BGR2RGB).ToWriteableBitmap();

                mask?.Dispose();
                result?.Dispose();
            });
        }

        Thread.Sleep(20);
    }
}

private Mat ChromaKeyMask(Mat src)
{
    Mat mask;
    using (var hsvMat = src.CvtColor(ColorConversionCodes.BGR2HSV))
    {
        var min = new Scalar(minH, minS, minV);
        var max = new Scalar(maxH, maxS, maxV);

        // Threshold the HSV image to extract green color
        mask = hsvMat.InRange(min, max);
        Cv2.BitwiseNot(mask, mask);

        // Smooth edges of the mask
        var size = new OpenCvSharp.Size(3, 3);
        var kernel = Cv2.GetStructuringElement(MorphShapes.Rect, size);
        Cv2.MorphologyEx(mask, mask, MorphTypes.Open, kernel);
        Cv2.GaussianBlur(mask, mask, size, 0, 0);
    }
    return mask;
}

This code is working but still not good enough for production grade: mask is "unstable" and "floating", and final image, after combining with background, has a green "halo".

app screenshot

So, my questions are:

  • how to make a mask more stable (I already tried contours based on mask but result isn't far away from the mask I'm getting by calling mask = hsvMat.InRange(min, max); ?
  • how properly combine video frame, mask and background image/video frame? Might be, alpha channel with transparency will help?

I'm really sorry (I'm new to OpenCV and image processing), but I'm asking for answers with specific working solutions, not general thoughts about chroma key (yes, I've already read the Wikipedia article and watched a lot of questions/answers here on StackOverflow) . There is no need to use C#; Python and C++ will work fine.

Here you can find a modified @Barak Python script with images for test.

I tried a different SDKs from https://www.visioforge.com and https://www.graphicsmill.com: pretty unstable and/or simple non-working.

I need a professional grade chroma key filtering solution, to process webcam stream in realtime.

0

There are 0 best solutions below