How to Manually Process Muxing h264 to mp4?

Question

The project I'm currently working on involves capturing screen frames on Windows via the Desktop Duplication API as raw BGRA format pixel data, and then encoding these frames into h264 format.

At this point, I have successfully created a video in the h264 format using the X264 crate. However, my next task is to mux the h264 frames into an mp4 file, and this is where I'm facing some difficulties. I am currently trying to utilize the ffmpeg-the-third crate, an FFmpeg wrapper in Rust, to perform this task, but I am unsure how to proceed with it.

I'm seeking assistance on how to use the ffmpeg-the-third crate or anything else that could allow me to convert the h264 video file to mp4 format. Specifically, I need help on how to implement the muxing process manually. Any guidance, examples, or resources that can guide me through this process would be greatly appreciated. Thank you!

Here is the code that I use to encode the captured frames into h264 via the x264 crate:

fn encode_frames(frames: Vec<Vec<u8>>, width: u32, height: u32, fps: u32) -> Result<()> {
    let mut file = File::create("../vid/captured_video.h264").unwrap();
    let mut encoder = Encoder::builder()
            .fps(fps, 1)
            .build(Colorspace::BGRA, width as _, height as _)
            .unwrap();
    {
        let headers = encoder.headers().unwrap();
        file.write_all(headers.entirety()).unwrap();
    }
    for (index, mut pic_data) in frames.into_iter().enumerate() {
        let image = Image::bgra(width as _, height as _, &pic_data);
        let (data, _) = encoder.encode((fps as usize * index) as _, image).unwrap();
        file.write_all(data.entirety()).unwrap();
    }
    // Flush out delayed frames
    {
        let mut flush = encoder.flush();
        while let Some(result) = flush.next() {
            let (data, _) = result.unwrap();
            file.write_all(data.entirety()).unwrap();
        }
    }
    Ok(())
}

Answer 1

I did this once using the minimp4 crate.

In your case it should look something like that (untested):

let mut mp4muxer = Mp4Muxer::new(File::create("my_file.mp4").unwrap());
let mut buf = Vec::new();
File::open("../vid/captured_video.h264").unwrap().read_to_end(&mut buf).unwrap();
mp4muxer.init_video(width, height, false, "title");
mp4muxer.write_video(&buf);
mp4muxer.close();

I also have this simple self-contained demonstration for encoding images to a mp4 video (in memory):

use std::io::{Cursor, Read, Seek, SeekFrom};

use image::{EncodableLayout, Rgb, RgbImage};
use imageproc::drawing::draw_filled_circle_mut;
use minimp4::Mp4Muxer;

use openh264::encoder::{Encoder, EncoderConfig};

fn main() {
    let config = EncoderConfig::new(512, 512);
    let mut encoder = Encoder::with_config(config).unwrap();

    let mut buf = Vec::new();

    for i in 0..512 {
        let frame = get_next_frame(i);
        // Convert RGB into YUV.
        let mut yuv = openh264::formats::RBGYUVConverter::new(512, 512);
        yuv.convert(&frame[..]);

        // Encode YUV into H.264.
        let bitstream = encoder.encode(&yuv).unwrap();
        bitstream.write_vec(&mut buf);
    }

    let mut video_buffer = Cursor::new(Vec::new());
    let mut mp4muxer = Mp4Muxer::new(&mut video_buffer);
    mp4muxer.init_video(512, 512, false, "Moving circle.");
    mp4muxer.write_video(&buf);
    mp4muxer.close();

    // Some shenanigans to get the raw bytes for the video.
    video_buffer.seek(SeekFrom::Start(0)).unwrap();
    let mut video_bytes = Vec::new();
    video_buffer.read_to_end(&mut video_bytes).unwrap();

    std::fs::write("circle.mp4", &video_bytes).unwrap();
}

fn get_next_frame(index: u32) -> Vec<u8> {
    let red = Rgb([255u8, 0u8, 0u8]);

    let mut image = RgbImage::new(512, 512);
    draw_filled_circle_mut(&mut image, (index as i32, index as i32), 40, red);

    image.as_bytes().to_vec()
}

with these dependencies:

image = "0.23.6"
imageproc = "0.22.0"
minimp4 = "0.1.0"
openh264 = "0.2.11"

Additional information on this topic can be found in this rustlang forum post.