Recording slow motion video using MediaCodec on Android

Question

I'm implementing video recording with Android camera. I use MediaCodec to encode frames from onPreviewFrame() callback.

Now I want to record slow motion video. How can I set the camera capture rate or frame rate? Where can I set these parameters, for example, if I want to record at 120fps, and play at 30fps?

The following is what I have researched:

I recorded a slow-motion video with MOTO X (having built-in slow-motion video recording function) and used FFmpeg to check its fps. I found it records at 110 fps(near 120fps) and playback at 30fps. This proves that it provides hardware support for recording at high fps.

However, when I use getSupportedPreviewFpsRange to check its supported fps range, there are only 4 sets supported fps range with no one greater than 30fps:

min_fps: 15.0 , max_fps15.0

min_fps: 15.0 , max_fps20.0

min_fps: 15.0 , max_fps30.0

min_fps: 24.0 , max_fps30.0

I used setPreviewFpsRange to set the fps higher, but it remains no more than 30fps. I also tried set KEY_FRAME_RATE and KEY_CAPTURE_RATE of MediaFormat. But it still does not work.

I have also searched the solution by setting CamcorderProfile or setVideoFrameRate with MediaRecorder. But I'm doing with MediaCodec.

Can anyone help me with this problem?

Answer 1

I think this question encloses two separate problems.

1. How can I activate a high-speed capture mode which is faster than in 60FPS?
2. How can I slow down the FPS generated by the MediaCodec encoder?

The second part is simple. As described in this answer, you can either insert interpolating/dummy frames, or modify time stamps. The first part is peculiar in that most sample codes deal with normal mode of capturing rather than high-speed captures required in this use case (usually higher than 60FPS, and generally speaking, we are dealing with 120~240FPS.)

About the first part of the question, I would like you to take a look at the description about Camera2 constrained high-speed capture APIs, like this. Unfortunately, there seem to be only niche demand on such an API and I was not able to gather much information upon my own findings, and I am currently dealing with mentally taxing projects myself to create a sample code by myself. However, judging from its similarity with ordinary movie recording and burst shots, I don't think deriving such a use case would be that much complex.

Answer 2

for slow recording.

mMediaRecorder.setVideoFrameRate(QUALITY_HIGH_SPEED_LOW);

for high speed recording.

mMediaRecorder.setVideoFrameRate(QUALITY_HIGH_SPEED_HIGH);

Answer 3

Capturing video frames at a higher rate doesn't address the problem. If you capture at 120fps, and play back at 120fps, you have high-speed video playing at a normal rate.

What you want to do is modify the timestamps. MediaCodec supports VFR (variable frame-rate) video. Rather than spending 1 second recording 60 frames that will be played back back at 30fps, you record 30 frames per second that will play back at 15fps.

The H.264 stream generated by MediaCodec's AVC encoder does not include timestamps at all. You specify the timestamp for each frame when you feed the output to MediaMuxer, generating the .mp4 file. Most sample code will simply take the timestamp obtained from Camera and pass it through MediaCodec to MediaMuxer unmodified, but you are allowed to tweak it. The only thing you can't do is allow the timestamp to go backward.

If you look at the way the "eight rects" movie is generated in Grafika, you can see that it plays with the output timestamp to make the movie play slower or faster.

Another easy way to accomplish the same thing without varying the frame rate is to double the frames. For the slow-motion portion, simply hand the same frame to the encoder multiple times. Again, you need to tweak the presentation timestamps for each frame, but your video will have a consistent 30fps. (It would look better if you interpolated frames, but that's a lot harder.)