Video recording and Frame processing at same time in Android using CameraX

Question

I am working on an app, my goal is to detect face, detect eyes blinking and recording video at the same time. So for this, I need to record video and at the same time I want to process frames to detect face and eyes blinking. I am using Camerax for this. There are some Android limitations for this case which are:

1. We cannot record video and use imageAnalysis methods of cameraX at the same time.
2. .setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL) will slow down the frame processing as it is also mentioned in MLKit documentation, so in my case as I need frame processing very fast, so I cannot use it too.

Dependencies which I am using:

// CameraX dependencies
def camerax_version = "1.2.0"
implementation "androidx.camera:camera-camera2:${camerax_version}"
implementation "androidx.camera:camera-lifecycle:${camerax_version}"
implementation "androidx.camera:camera-view:${camerax_version}"
implementation "androidx.camera:camera-video:${camerax_version}"
// Google mMLKit to detect faces (unbundled)
implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0'

Preview View in xml file:

        <androidx.camera.view.PreviewView
        android:id="@+id/previewView"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

Setting camera:

@Nullable
protected ProcessCameraProvider cameraProvider;
@Nullable
private Preview previewUseCase;
private VideoCapture<Recorder> videoCapture;
private CameraSelector cameraSelector;

private void initCamera() {
    try {
        cameraSelector = new CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_FRONT).build();
        new ViewModelProvider(this, (ViewModelProvider.Factory)
                ViewModelProvider.AndroidViewModelFactory.getInstance(getApplication()))
                .get(CameraXViewModel.class)
                .getProcessCameraProvider()
                .observe(getViewLifecycleOwner(),
                        provider -> {
                            cameraProvider = provider;
                            bindAllCameraUseCases();
                        });
    } catch (Exception e) {
        
    }
}

/**
 * method to init binding camera use cases
 * will check camera provider, will unbind previous use cases
 * and will call further method to bind camera use cases
 */
private void bindAllCameraUseCases() {
    try {
        if (cameraProvider != null) {
            cameraProvider.unbindAll();
            if (cameraProvider == null) {
                return;
            }
            if (previewUseCase != null) {
                cameraProvider.unbind(previewUseCase);
            }
            bindCameraUseCases();
        }
    } catch (Exception e) {
        
    }
}

/**
 * method to bind camera use cases
 * will get camera provider instance
 * and will bind preview and video use case
 * invoking further method to set FaceMesh detection
 */
private void bindCameraUseCases() {
    try {
        Preview.Builder builder = new Preview.Builder();
        previewUseCase = builder.build();
        previewUseCase.setSurfaceProvider(fragmentCameraBinding.previewView.getSurfaceProvider());
        Recorder recorder = new Recorder.Builder()
                .setQualitySelector(QualitySelector.from(Quality.LOWEST, FallbackStrategy.higherQualityOrLowerThan(Quality.LOWEST))).build();
        videoCapture = VideoCapture.withOutput(recorder);
        new ViewModelProvider(this, (ViewModelProvider.Factory)
                ViewModelProvider.AndroidViewModelFactory.getInstance(getActivity().getApplication()))
                .get(CameraXViewModel.class)
                .getProcessCameraProvider()
                .observe(
                        getViewLifecycleOwner(),
                        provider -> {
                            cameraProvider = provider;
                            try {
                                cameraProvider.unbindAll();
                                try {
                                    Camera cameraX = cameraProvider.bindToLifecycle(getViewLifecycleOwner(), cameraSelector,
                                            previewUseCase, videoCapture);
                                } catch (Exception e) {
                                }
                            } catch (Exception e) {
                            }
                        });
    } catch (Exception e) {
    }
}

As I mentioned that, I cannot use ImageAnalysis method to get frames, so I am getting frames manually like this:

Bitmap bitmap = fragmentCameraBinding.previewView.getBitmap();

and then I am using this bitmap for face detection and to detect eyes blinking. While at the same time I am recording video too, code for this:

@Nullable
protected String storagePath = "";
private VideoCapture<Recorder> videoCapture;
private Recording currentRecording;
private File videoFile;
private final Consumer<VideoRecordEvent> videoCallback = new Consumer<androidx.camera.video.VideoRecordEvent>() {
    @Override
    public void accept(androidx.camera.video.VideoRecordEvent videoRecordEvent) {
        try {
            if (videoRecordEvent instanceof VideoRecordEvent.Start) {
                //video recording started
            } else if (videoRecordEvent instanceof VideoRecordEvent.Finalize) {
                //video recording stopped
            }
        } catch (Exception e) {
        }
    }
};
private void startRecording() {
    try {
        long timeStamp = System.currentTimeMillis();
        ContentValues contentValues = new ContentValues();
        contentValues.put(MediaStore.MediaColumns.DISPLAY_NAME, timeStamp);
        contentValues.put(MediaStore.MediaColumns.MIME_TYPE, "video/mp4");

        File directory = requireContext().getCacheDir();
        videoFile = null;
        try {
            videoFile = File.createTempFile(
                    "recorded_file",
                    ".mp4",
                    directory
            );
        } catch (IOException e) {
            e.printStackTrace();
        }
        if (videoFile != null) {
            storagePath = videoFile.getPath();
            FileOutputOptions fileOutputOptions = new FileOutputOptions.Builder(videoFile).build();
            currentRecording = videoCapture.getOutput().prepareRecording(requireActivity(), fileOutputOptions).start(
                    getExecutor(), videoCallback);
        }
    } catch (Exception e) {
    }
}

private Executor getExecutor() {
    return ContextCompat.getMainExecutor(context);
}

To stop video recording:

currentRecording.stop();

previewView.getBitmap, as it is related to UI, so I have to do it on the main UI thread. It takes time, further, face detection is also taking time. Overall, My code is processing 3 to 4 frames per second, I want to process 16-18 frames per second, as devices is returning almost 24-25 frames per second.

Answer 1

CameraX supports concurrent VideoCapture and ImageAnalysis on device with camera hardware level LEVEL_3. On lower level devices, I believe if you use the latest CameraX release(1.3.0-rc02), it also supports concurrent VideoCapture and ImageAnalysis. However, it has a cost. Internally CameraX uses OpenGL to copy frames to both Preview and VideoCapture. So you might see a degradation on performance and system health.

As you mentioned, PreviewView#getBitmap is inefficient. We don't recommend that.

MLKit's face detection being slow on the other hand, is a harder problem to solve. If it's too slow, you may consider skipping frames.