How to manipulate CameraPreview bytearray through the JNI? (OpenCV)

Question

Inside my "public void onPreviewFrame(byte[] data, Camera camera) { .... }" I want to take the bytearray "data" and pass it to the JNI and apply some OpenCV filters so that the preview changes, without returning it. What is the best way to do this?

Currently I only pass the bytearray like this:

JNIEXPORT jint JNICALL Java_example_jnitest_Lib_filterfunc
      (JNIEnv * je, jclass jc, jbyteArray byteData){
    try {
         jbyte* _b_data= je->GetByteArrayElements(byteData, 0);
         int height = base;
         int width = base2;
         Mat mdata(height, width, CV_8UC4, (unsigned char *)_b_data);
         Mat myMat = imdecode(mdata,1);
         je->ReleaseByteArrayElements(byteData, _b_data, 0);
         return 1;
    } catch(const exception& ex){
        return 0;
    }
}

In the Java Code:

 public static native int filterfunc(byte[] byteData);

Note: Currently the return Value is one so I hope turning the bytearray into a Mat Obejct is working. But this won't change the Preview because I don't change it back to an bytearray.

Answer 1

I want to take the bytearray "data" and pass it to the JNI and apply some OpenCV filters so that the preview changes, without returning it.

Unfortunately that's not possible. The byte array that is passed to onPreviewFrame() is just a copy of the preview frame, and any changes that you make to it will not be shown in the preview. You can test this for yourself by modifying the byte array in Java inside the onPreviewFrame() function as a test, you won't see any effect.

If you want to change the preview frame data using OpenCV and see the results in a preview window then you will need to upload the processed frame to an OpenGL texture and then render it to a GLSurfaceView, using a fragment shader to convert the NV21 data to RGB, or some other approach. Simply changing the byte array won't work.

See these questions for more information:

PreviewCallback onPreviewFrame does not change data

onPreviewFrame doesn't change the data

Answer 2

You can use OpenCV implementation of accessing camera.

Here is a sample code using OpenCV4android.

public class SampleCameraFrameAccessActivity extends Activity implements        CvCameraViewListener2, OnTouchListener{
private static final String  TAG = "SampleCameraFrameAccessActivity";

protected CameraBridgeViewBase cameraPreview;
protected Mat mRgba;

protected BaseLoaderCallback  mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
            case LoaderCallbackInterface.SUCCESS:
            {
                Log.i(TAG, "OpenCV loaded successfully");
//                    mOpenCvCameraView.enableView();
//                    mOpenCvCameraView.setOnTouchListener(ColorRegionDetectionActivity.this);
                cameraPreview.enableView();
            } break;
            default:
            {
                super.onManagerConnected(status);
            } break;
        }
    }
};

@Override
protected void onCreate(Bundle savedInstanceState) {
    // TODO Auto-generated method stub
    super.onCreate(savedInstanceState);
    setContentView(R.layout.camera_sample_layout);

    cameraPreview = (CameraBridgeViewBase) findViewById(R.id.sample_test_camera_view);

    cameraPreview.setCvCameraViewListener(this);

}

@Override
protected void onDestroy() {
    // TODO Auto-generated method stub
    super.onDestroy();
}

@Override
protected void onPause() {
    // TODO Auto-generated method stub
    super.onPause();
    if(cameraPreview != null){
        cameraPreview.disableView();
    }
}

@Override
protected void onResume() {
    // TODO Auto-generated method stub
    super.onResume();
    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback);
}

@Override
public void onCameraViewStarted(int width, int height) {
    // TODO Auto-generated method stub
    mRgba =  new Mat(height, width, CvType.CV_8UC4);
}

@Override
public void onCameraViewStopped() {
    // TODO Auto-generated method stub
    mRgba.release();

}

@Override
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    // TODO Auto-generated method stub
    mRgba = inputFrame.rgba();

    return mRgba;
}

@Override
public boolean onTouch(View v, MotionEvent event) {
    // TODO Auto-generated method stub
    return false;
}

}

And the XML Layout file is :

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:id="@+id/sample_test_layout" >

    <org.opencv.android.JavaCameraView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:id="@+id/sample_test_camera_view" />

</RelativeLayout>

In onCameraFrame method you can access every frame from the frame buffer of the camera. if you want to apply opencv filters you can simply pass the frame buffer to jni or use opencv4android wrappers to do it.