Is it possible to show video face detection in the capture stream?

Question

I am using the MediaCapture API for a video recording app.

Part of it is face detection using rectangles, but I would like that to appear in the capture stream, not just the preview stream.

Even though I indicate VideoRecord as the MediaStreamType when adding the effect, the rectangles do not appear in the end-result video, which is my goal.

var definition = new FaceDetectionEffectDefinition
 {
    SynchronousDetectionEnabled = false,
    DetectionMode = FaceDetectionMode.HighPerformance
};

try
{
    faceDetectionEffect = (FaceDetectionEffect)await mediaCapture.AddVideoEffectAsync(definition, MediaStreamType.VideoRecord);
}
catch (Exception exception)
{
    // Error message
    return;
}
faceDetectionEffect.FaceDetected += FaceDetectionEffect_FaceDetected;
faceDetectionEffect.DesiredDetectionInterval = TimeSpan.FromMilliseconds(33);
faceDetectionEffect.Enabled = true;

Any advice on how to cause face detection and tracking to appear in the capture stream, or the end-result video, as well?

Thanks!

==================================

Basically, I would like face detection and tracking to appear in the CaptureElement so that identity hiding (obscuring the user's face with an image that tracks their face) can be accomplished in the final video.

My code borrows from the Windows universal samples: https://github.com/microsoft/Windows-universal-samples

There is a CaptureElement where the video stream from the camera is rendered. To draw the rectangles of the detected faces, a Canvas is used.

<CaptureElement Name="PreviewControl"                            
                            Opacity="0.8"
                            Stretch="Fill"
                            IsDoubleTapEnabled="True" />

<Canvas>
    <Canvas Name="FacesCanvas"
        RenderTransformOrigin="0.5,0.5" />
                </Canvas>

This is how the FaceDetectionEffectDefinition is added to the MediaCapture object, and the FaceDetectionEffect is added...

private async Task CreateFaceDetectionEffectAsync()
{
    var definition = new FaceDetectionEffectDefinition
    {
            SynchronousDetectionEnabled = false,
            DetectionMode = FaceDetectionMode.HighPerformance
    };
    try
    {
            faceDetectionEffect = (FaceDetectionEffect)await mediaCapture.AddVideoEffectAsync(definition,       MediaStreamType.VideoRecord);
    }
    catch (Exception exception)
    {
            //exception.Message.ToString()
            return;
    }
    faceDetectionEffect.FaceDetected += FaceDetectionEffect_FaceDetected;
    faceDetectionEffect.DesiredDetectionInterval = TimeSpan.FromMilliseconds(33);
    faceDetectionEffect.Enabled = true;
}

What goes on inside the FaceDetectionEffect event...

private async void FaceDetectionEffect_FaceDetected(FaceDetectionEffect sender, FaceDetectedEventArgs args)
{
    await MainPage.Current.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => HighlightDetectedFaces(args.ResultFrame.DetectedFaces));
}

private void HighlightDetectedFaces(IReadOnlyList<DetectedFace> faces)
{
    facesCanvas.Children.Clear();
    if (softwareBitmapSource == null)
        return;

                for (int i = 0; i < faces.Count; i++)
                {
                    Rectangle faceBoundingBox = ConvertPreviewToUiRectangle(faces[i].FaceBox);                
                    faceBoundingBox.Fill = new ImageBrush
                    {
                            ImageSource = softwareBitmapSource
                    };                
                    facesCanvas.Children.Add(faceBoundingBox);
                }               
}

The MediaCapture object get initialized, and is set as the source of the CaptureElement in the beginning...

(Some code, such as specifying the media stream type for VideoPreview and VideoRecord, if not identical, has been omitted.)

MediaCaptureInitializationSettings settings = new MediaCaptureInitializationSettings { VideoDeviceId = preferredCamera.Id, AudioDeviceId = preferredMicrophone.Id };

await mediaCapture.InitializeAsync(settings);

PreviewControl.Source = mediaCapture;

Hopefully, there is an easier technique, without having to resort to using MediaComposition and overlays.

Answer 1

You may need to create a video effect to get the detected face rectangle added as part of the video capture. What this will allow for is access to the raw bitmap data in each media sample where you can then draw whatever you need.

FaceDetector may be better than FaceTracker, but you will lose the predictive nature of the later. You could try to refer to the document which may be helpful.