inputImage.getByteBuffer() is null when using AndroidX Camera with MLKit

Question

This is the code from my Activity class:

    @Override protected void onCreate(Bundle savedInstanceState)
    {
        requestWindowFeature(Window.FEATURE_NO_TITLE);
        super.onCreate(savedInstanceState);

        _cpf = ProcessCameraProvider.getInstance(this);
        _cpf.addListener(new Runnable() {
            @Override
            public void run() {
                try {
                    ProcessCameraProvider cameraProvider = _cpf.get();
                    bindImageAnalysis(cameraProvider);
                } catch (ExecutionException | InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }, ContextCompat.getMainExecutor(this));
    }

    private void bindImageAnalysis(@NonNull ProcessCameraProvider cameraProvider) {
        ImageAnalysis imageAnalysis =
                new ImageAnalysis.Builder().setTargetResolution(new Size(640, 360))
                        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build();
        imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this), new ImageAnalysis.Analyzer() {
            @Override
            public void analyze(@NonNull ImageProxy ip) {
                processImage(ip.getImage(), ip.getImageInfo().getRotationDegrees());
                ip.close();
            }
        });
        CameraSelector cameraSelector = new CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
        cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, imageAnalysis);
    }

On every analyze callback, MLKit indicates that InputImage.fromMediaImage(ip.getImage(),ip.getImageInfo().getRotationDegrees()).getByteBuffer() is null. How should I interpret this? Does this always mean that the image is null?

Answer 1

The problem was in the fact that I was .close()ng prematurely. PoseDetector processing is asynchronous so processImage is being called and immediately after ip.close(). When ip.close() is called, processImage is still working so this is the reason behind Attempt to invoke virtual method 'java.nio.ByteBuffer android.media.Image$Plane.getBuffer(). Instead of using .close() inside the analyze method, it should be invoked at the end of OnSuccessListener or OnSuccessListener.

Here is the full example.

Answer 2

inputImage.getByteBuffer() is an internal method which is not designed for public usage. This method will only return when you use a bytebuffer or byte array to construct the image. If you need to use a bytebuffer from your android media image, you will need to do the conversion on your own.

Also, we made some changes to Firebase ML Kit to better distinguish the on-device APIs from cloud based APIs. "ML Kit"(without firebase branding) contains all the on-device APIs. Here's the migration guide from firebase mlkit to mlkit. All further improvements and new APIs will be released only with the new ML Kit.