Video with alpha channel overlay on background image: Alpha shows black

Question

I need to play a .mov video (ProRes4444) with alpha channel in a scene. The scene has a background image and I need to use the alpha channel of the video so it overlays on the background.

If I open the video normally with QMediaPlayer, the alpha channel appears in black.

screen with background pic & video with black alpha:

How can I make the output of the QMediaPlayer (QGraphicsVideoItem) respect the alpha and make the overlay effect possible?

The closest I got to the answer based on online research is code in cpp that I've found that shows the necessity to create a subclass of a QAbstractVideoSurface that receives videoframes converts to ARGB, then forwards those to a QLabel that displays them.

Displaying a video with an alpha channel using qt

I've also tried that unsuccessfully. Is this the right course or I'm just missing something simple on my current code?

EDIT: Link to files (background image and video .mov) https://drive.google.com/drive/folders/1LIZzTg1E8wkaD0YSvkkcfSATdlDTggyh?usp=sharing

import sys
from PyQt5.QtMultimedia import *
from PyQt5.QtMultimediaWidgets import *
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *
from PyQt5.QtGui import *


class VideoWindow(QMainWindow):
     def __init__(self):
        super(VideoWindow, self).__init__()
        self.setWindowTitle('QMediaPlayer TEST')
        self.resize(1920, 1080)

        self.vista = QGraphicsView(self)
        self.vista.setGeometry(QRect(0, 0, 1920, 1080))

        self.scene = QGraphicsScene(self.vista)
        self.scene.setSceneRect(0, 0, 1920, 1080)
        self.vista.setScene(self.scene)

        self.graphvitem1 = QGraphicsVideoItem()

#SET BACKGROUND IMAGE ON SCENE
        self.tempImg = QPixmap("/Users/elemental/Desktop/pyvids/fons.jpeg")
        self.tempImg = self.tempImg.scaled(self.scene.width(), self.scene.height())
        self.graphicsPixmapItem = QGraphicsPixmapItem(self.tempImg)
        self.scene.addItem(self.graphicsPixmapItem)

#SET VIDEO 1 WITH LOOP
        self.mediaPlayer1 = QMediaPlayer(None, QMediaPlayer.VideoSurface)
        self.mediaPlayer1.setVideoOutput(self.graphvitem1)

        self.playlist1 = QMediaPlaylist(self)
        self.playlist1.addMedia(QMediaContent(QUrl.fromLocalFile("/Users/elemental/Desktop/pyvids/vida1.mov")))
        self.playlist1.setCurrentIndex(1)
        self.playlist1.setPlaybackMode(QMediaPlaylist.CurrentItemInLoop)
        self.mediaPlayer1.setPlaylist(self.playlist1)

        self.graphvitem1.setPos(500, 100)
        self.graphvitem1.setSize(QSizeF(1000, 500))
        self.scene.addItem(self.graphvitem1)

        self.mediaPlayer1.play()
        self.vista.show()


if __name__ == '__main__':
     app = QApplication([])
     window = VideoWindow()
     window.show()
     sys.exit(app.exec_())

Answer 1

From what I can see, QVideoWidget doesn't support alpha channels by default, so it falls back to the "basic" black background.

But, implementation seems possible, by properly subclassing QAbstractVideoSurface.

Consider that the following code is experimental, my knowledge of QMediaPlayer and the Qt video surface isn't that deep (the former is an abstract for multiple platforms and multiple libraries that can behave very differently on different configurations), and I could only test it on two Linux platforms, so I don't know how it behaves under Windows nor MacOS.

The assumption is that the video surface provides a default dedicated QWidget subclass (VideoWidget) unless another class with a suitable setImage is provided, and updates its image whenever the media player requires it.

Note that I only tested it with a couple of videos (including the provided one), and further testing might be required.

from PyQt5.QtMultimedia import *
from PyQt5.QtMultimediaWidgets import *
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *
from PyQt5.QtGui import *

class VideoWidget(QWidget):
    image = QImage()
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def setImage(self, image):
        self.image = image
        self.update()

    def sizeHint(self):
        return QSize(640, 480)

    def paintEvent(self, event):
        qp = QPainter(self)
        # ensure that smooth transformation is used while scaling pixmaps
        qp.setRenderHints(qp.SmoothPixmapTransform)

        # provide compliancy with background set using stylesheets, see:
        # https://doc.qt.io/qt-5/stylesheet-reference.html#qwidget-widget
        opt = QStyleOption()
        opt.initFrom(self)
        self.style().drawPrimitive(QStyle.PE_Widget, opt, qp, self)

        # draw the image, scaled to the widget size; if you need fixed sizes
        # or keep aspect ratio, implement this (or the widget) accordingly
        qp.drawImage(self.rect(), self.image, self.image.rect())


class AlphaVideoDrawer(QAbstractVideoSurface):
    def __init__(self, videoWidget=None, widgetOptions=None):
        super().__init__()
        if videoWidget:
            if not hasattr(videoWidget, 'setImage'):
                raise NotImplementedError(
                    'setImage() must be implemented for videoWidget!')
        else:
            if not isinstance(widgetOptions, dict):
                widgetOptions = {}
            elif not 'styleSheet' in widgetOptions:
                # just a default background for testing purposes
                widgetOptions = {'styleSheet': 'background: darkGray;'}
            videoWidget = VideoWidget(**widgetOptions)
        self.videoWidget = videoWidget

        # QVideoFrame.image() has been introduced since Qt 5.15
        version, majVersion, minVersion = map(int, QT_VERSION_STR.split('.'))
        if version < 6 and majVersion < 15:
            self.imageFromFrame = self._imageFromFrameFix
        else:
            self.imageFromFrame = lambda frame: frame.image()

    def _imageFromFrameFix(self, frame):
        clone_frame = QVideoFrame(frame)
        clone_frame.map(QAbstractVideoBuffer.ReadOnly)
        image = QImage(
            clone_frame.bits(), frame.width(), frame.height(), frame.bytesPerLine(), 
            QVideoFrame.imageFormatFromPixelFormat(frame.pixelFormat()))
        clone_frame.unmap()
        return image

    def supportedPixelFormats(self, type):
        return [QVideoFrame.Format_ARGB32]

    def present(self, frame: QVideoFrame):
        if frame.isValid():
            self.videoWidget.setImage(self.imageFromFrame(frame))

        if self.surfaceFormat().pixelFormat() != frame.pixelFormat() or \
            self.surfaceFormat().frameSize() != frame.size():
                self.setError(QAbstractVideoSurface.IncorrectFormatError)
                self.stop()
                return False
        else:
            return True


class AlphaVideoTest(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setStyleSheet('''
            QFrame#mainFrame {
                background: green;
            }
        ''')

        mainFrame = QFrame(objectName='mainFrame')
        self.setCentralWidget(mainFrame)

        layout = QVBoxLayout(mainFrame)
        self.playButton = QPushButton('Play', checkable=True)
        layout.addWidget(self.playButton)

        self.drawer = AlphaVideoDrawer()
        layout.addWidget(self.drawer.videoWidget)

        self.mediaPlayer1 = QMediaPlayer(self, QMediaPlayer.VideoSurface)
        self.playlist = QMediaPlaylist(self)
        path = QDir.current().absoluteFilePath('vida1.mov')
        self.playlist.addMedia(QMediaContent(QUrl.fromLocalFile(path)))
        self.playlist.setCurrentIndex(1)
        self.playlist.setPlaybackMode(QMediaPlaylist.CurrentItemInLoop)

        self.mediaPlayer1.setPlaylist(self.playlist)
        self.mediaPlayer1.setVideoOutput(self.drawer)

        self.playButton.toggled.connect(self.togglePlay)

    def togglePlay(self, play):
        if play:
            self.mediaPlayer1.play()
            self.playButton.setText('Pause')
        else:
            self.mediaPlayer1.pause()
            self.playButton.setText('Play')

import sys
app = QApplication(sys.argv)
test = AlphaVideoTest()
test.show()
sys.exit(app.exec_())

I based the above code on the following sources:

• wrong video frame of present() of qabstractvideosurface in pyqt5
• Displaying a video with an alpha channel using qt

Note that I limited the supportedPixelFormats output, as using the full list of formats provided in the related question didn't work; this doesn't mean that this would work anyway, but that further testing is probably required, possibly on different machines and different OS/System configurations and video formats: remember that QMediaPlayer completely relies on the underlying OS and default media backend.

Finally, if you only need this for "limited" and predefined animations, consider implementing your own subclass of QWidget that uses a list of loaded PNG images and shows them by implementing paintEvent() that would be called by updates based on a QVariantAnimation. While this kind of implementation might result less performant or ideal, it has the major benefit of providing cross platform compatibility.