Following the MediaPipe Pose landmark detection guide for the web (that you can find here) I managed to correctly display and overlay the resulting landamarks on a video. But I am struggling to display the segmentation mask (which should be possible according to this demo.
After setting the option outputSegmentationMasks to true, I run the Task using PoseLandmarker.detectedVideo(). The attribute segmentationMasks is available and returns a list of poses (I guess), and result[0].canvas is an instance of OffscreenCanvas.
I was hoping to draw the resulting mask using
// rendering loop
canvasCtxVideo.save();
canvasCtxVideo.clearRect(0, 0, canvasCamera.clientWidth, canvasCamera.height);
canvasCtxVideo.drawImage(
result.segmentationMasks[0].canvas,
0,
0,
canvasCamera.clientWidth,
canvasCamera.height
);
canvasCtxVideo.restore();
but nothing is shown on the HTMLCanvas.
How can I achieve this?
UPDATE:
Since result[0].canvas is an OffsetCanvas I started looking into the Canvas API. This article suggests that you can either use a WebWorker, or convert the canvas to a an ÌmageBitmap using the method OffsetBuffer.transferToImageBitmap().
Unfortunately neither solution works, because:
- it is not possible to transfer an
OffsetCanvasto aWebWorkerif it already has a rendering context. - with the following snippet the canvas remains blank:
const maskCanvas = document.getElementById("mask_canvas")
const maskCanvasCtx = maskCanvas.getContext("bitmaprenderer")
const mask = result.segmentationMasks[0]
const image = mask.canvas.transferToImageBitmap()
maskCanvasCtx.transferFromImageBitmap(image)