Why pyav package in python can not recognize h264_cuvid codec while ffmpeg can do?

Question

I have a python script as below to read byte_stream from video. I want to use h264_cuvid code to do GPU hardware accelerating instead of CPU encoding or decoding. my FFmpeg does have h264_cuvid codecs when I run this command:

ffmpeg -c:v h264_cuvid -i input.mp4 output_codec.mp4

it runs successfully but in pyav package, when I tried to create a h264_cuvid codec as below:

import av

video = av.open(VIDEO_FILE_PATH)
target_stream = video.streams.video[0]
ctx = av.Codec('h264_cuvid', 'r').create()

it gave me an error which said an unknown codec:

ctx = av.Codec('h264_cuvid', 'r').create()
File "av/codec/codec.pyx", line 184, in av.codec.codec.Codec.__cinit__
File "av/codec/codec.pyx", line 193, in av.codec.codec.Codec._init
av.codec.codec.UnknownCodecError: h264_cuvid

av version = 9.0.2

ffmpeg version 7:3.4.8-0ubuntu0.2

Note: I installed av by pip install av

any help would be appreciated

Answer 1

Unless you compile pyAV from source it will come precompiled with a vendored version of ffmpeg that is built without support for hardware acceleration. This is done for two reasons (1) better device compatibility, and (2) efficiency of maintenance.

People typically cite three major reasons for using hwaccel: (1) faster encoding, (2) faster decoding, or (3) better energy efficiency. Regarding (1), hardware encoders tend to generate lower quality video than software encoders in practice (bitrate equated). Regarding (2), hardware decoders are indeed faster than software decoders but will have to copy the frame back into host memory (RAM) to make it available to the python layer. This will diminish or nullify any speed gains. Regarding (3), if you are after energy efficiency then switching to a compiled language would be a much better optimization than introducing hwaccel.

The main reason against hwaccel is: It's hard to implement and brittle, so not only do you need a quite talented individual to write the initial code, but you also need an equally capable person to volunteer to maintain it and fix things once the hwaccel API changes (it isn't the most stable) or if there are bugs or new features to add.

So in sum, the main reason why it isn't supported in pyav is because it is seen as not impactful enough, and - more importantly - because there isn't anyone willing to maintain it, i.e., it isn't supported for efficiency of maintainance.

Answer 2

I have answered my own question because I found the solution. In order to use hardware accelerating in pyav, you need to compile ffmpeg and then install pyav from the source (not using pip install pyav). Also you have to set cuda path of your system into your bashrc file

this bashscript helps one to do all steps:

echo "************************  try building ffmpeg  *******************"
echo "Install nv-codec-headers"

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git

cd nv-codec-headers && make -j4 && make install && cd ..

wget https://ffmpeg.org/releases/ffmpeg-4.2.1.tar.bz2
tar -xf ffmpeg-4.2.1.tar.bz2
cd ffmpeg-4.2.1
./configure --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --enable-shared

make -j4
make install
cd ..

echo "************************  setting cuda path  *******************"
echo "export PATH=/usr/local/cuda/bin:/ffmpeg/libavdevice:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/cuda/targets/x86_64-linux/include:/usr/local/cuda/lib64:/usr/local/cuda/include:/usr/local/cuda/extras/CUPTI/lib64:/ffmpeg/libavdevice" >> ~/.bashrc

echo "************************  try building pyav  *******************"
git clone -b hwaccel https://github.com/rvillalba-novetta/PyAV.git

cd PyAV
source scripts/activate.sh
pip3 install -r tests/requirements.txt
make -j4

python3 setup.py install
cd ..

ldconfig /usr/local/cuda/lib64
echo "*********************   ffmpeg and pyav are built successfuly ***************"