generate video with PyAv (from image generator and audio file)

2.3k Views Asked by At

There is a logic based on the python generator that uses PyAv to create a fixed sound file and an image equal to the length or 1 frame longer. However, I want to perform this process at once, and I want to implement it without calling ffmpeg additionally.

I know about demuxing sound and video files and muxing them via PyAv, but I can't save the video copy of the file I'm trying to convert with the final product due to insufficient system storage space.

So, how mux audio packet and video frame?

import av
import numpy as np

def video_generator(config):
    for _ in range( int(<audio time by second> * config.fps)):
        yield np.zeros((config.height, config.width, 3), dtype = np.uint8)

output = av.open(config.output_file, 'w')
ovstream = output.add_stream('libx264rgb', config.fps)
ovstream.pix_fmt = 'rgb24'
ovstream.width = config.width
ovstream.height = config.height

audio_input = av.open(<input wav>, 'r')
oastream = output.add_stream('mp3')

v_frames = (av.VideoFrame.from_ndarray(img, format='rgb24') for img in video_generator(config))
v_frames = (ovstream.encode(frame) for frame in v_frames)

for img_packet in v_frames:
    output.mux(img_packet)
output.close()
1

There are 1 best solutions below

0
On
...
for img_packet in v_frames:
    output.mux(img_packet)

stream = audio_input.streams.audio[0]
for packet in audio_input.demux((stream,)):
    for frame in packet.decode():
        a_frames = oastream.encode(frame)
        output.mux(a_frames)
output.close()
...

Also, if frame drops occur, Adding output.mux(ovstream.encode()); output.mux(oastream.encode()) right before output.close()

https://github.com/PyAV-Org/PyAV/issues/597#issuecomment-575349451