Android Camera2: Update UI between image captures with captureBurst is not synchronized

Question

I am trying to capture images with Camera2 API on Android with captureBurst() method. I start the burst capture in my button's OnClickListener:

button?.setOnClickListener(View.OnClickListener {
            imageRequestBuilder?.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE)
            imageRequest = imageRequestBuilder?.build()
            val requests = Collections.nCopies(10, imageRequest)
            cameraCaptureSession?.captureBurst(requests, captureCallback, backgroundHandler)
        })

In my OnImageAvailableListener listener, I am then reading the captured image, saving it and closing. At this point I would like to display something on my screen and I would like to it in between every captured image in the burst sequence. So I have written this code:

override fun onImageAvailable(reader: ImageReader?) {
    val image = reader?.acquireNextImage()
    if(image!=null) {
        synchronized(uiUpdate) {
            runOnUiThread(uiUpdate)
            (uiUpdate as java.lang.Object).wait()
        }

        val buffer = image?.planes?.get(0)?.buffer
        val bytes = ByteArray(buffer?.capacity()!!)
        buffer.get(bytes)
        SaveImageTask().execute(bytes)
    }

    image?.close()
}

private val uiUpdate = object : Runnable {
    override fun run() {
        synchronized(this) {
            textView?.text = "" + ++imageNum
            (this as java.lang.Object).notify()
        }
    }
}

To test it and to be sure that this is realy happening, I am using my front camera, and holding a mirror above my device so that camera actually captures the app's display. Unfortunately, the display and captures are not synchronized. I have triend to put 10 same ImageRequests to captureBurst method, but when I overview the obtained images, first three are the same (display is still displaying initial number 0), but the rest are fine (display is changing synchronosly). Why can't all be synchronized, i.e. why are first few images captured almost at the same time (at the start of the burst), and others are fine?

I have created my ImageReader with maximum of 1 Image in order to not to capture more than one image at the time:

imageReader = ImageReader.newInstance(/*some width*/, /*some height*/, /*some format*/, 1)

And the RequestBuilder is created with TEMPLATE_PREVIEW mode in order to maximize capturing screen.

I am aware that I could call capture method every time I capture new image in my OnImageAvailableListener listener, and this is working (tested!). But I need to get my application to capture images in fastest time possible.

Any help?

EDITED: This is my log as @alex-cohn has suggested:

D/onImageAvailable: display #0 Timestamp #0: 111788230978655
D/onImageAvailable: display #1 Timestamp #1: 111788264308655
D/onImageAvailable: display #2 Timestamp #2: 111788297633655
D/onImageAvailable: display #3 Timestamp #3: 111788730892655
D/onImageAvailable: display #4 Timestamp #4: 111788930856655
D/onImageAvailable: display #5 Timestamp #5: 111789030840655
D/onImageAvailable: display #6 Timestamp #6: 111789097494655
D/onImageAvailable: display #7 Timestamp #7: 111789264133655
D/onImageAvailable: display #8 Timestamp #8: 111789364112655
D/onImageAvailable: display #9 Timestamp #9: 111789464097655

EDITED2:

I have managed to get one image per one displayed number, but I am dropping 2 images every time after I had captured one, i.e.:

private val onImageAvailableListener = ImageReader.OnImageAvailableListener {reader ->
        val image = reader?.acquireLatestImage()

        if(image!=null) {
            capturedImagesCount++

            if(capturedImagesCount % frameDrop == 1) {
                Log.i("IMAGE_TIME", "Image available at: "+ SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS", Locale.getDefault()).format(Date()))
                synchronized(uiUpdate) {
                    runOnUiThread(uiUpdate)
                    (uiUpdate as java.lang.Object).wait()
                }

                val buffer = image.planes?.get(0)?.buffer
                val bytes = ByteArray(buffer?.capacity()!!)
                buffer.rewind()
                buffer.get(bytes)
                SaveImageTask().execute(bytes)
            }
        }

        image?.close()
    }

here, the frameDrop is set to 3. Also, maxImages in creating the imageReader is also set to 3. I had tested it on the one other device which is taking 2 images at the time, so in that device's case I had to set frameDrop and maxImages to 2. The downside of this is that it is still a bit slow, after all it is taking every third(second) image taken in burst mode which is not exctly the thing you want when using captureBurst method.

I still do not quite understand why this is working this way and why is camera taking images in something like pairs or triplets.

Answer 1

Setting ImageReader.maxImages to 1 does not help, because it only effects the way your app receives the images from camera (it will only receive one at a time), but this does not give you full control over the internal behaviour of the camera.

When the burst session starts, the camera (outside your app process) loads as much as possible (in you case, 3 images), while passing the images to your app (across process boundaries) one at a time. This cross-process work takes time, and only after that you have a chance to wait on the camera callback thread for UI update.

You are lucky to have caught this behaviour on your development device. On other devices, the camera internals may be different, and it may only buffer one image (as you would expect), or even more than 3. This number may also depend on the image resolution.

If you desperately need to tightly synchronize the images with UI, you need a way to drop first few images. How many? You may need to check for each device and image size.

You may find that ImageReader.acquireLatestImage() (having set maxImages to 3) will work better for you.

To have better control over the process, you could have something like

private var imageConsumedCount = 0
private var imageDisplayedCount = 0
override fun onImageAvailable(reader: ImageReader?) {
    val image = reader?.acquireNextImage()
    if (image!=null) {
        Log.d("onImageAvailable", "display #${imageDisplayedCount} Timestamp #${imageConsumedCount}: ${image?.timestamp}")
        imageConsumedCount++
        val next_image = reader?.acquireNextImage()
        while (next_image != null) {
            imageConsumedCount++
            image.close()
            image = next_image
            next_image = reader?.acquireNextImage()
        }
        synchronized(uiUpdate) {
            runOnUiThread(uiUpdate)
            (uiUpdate as java.lang.Object).wait()
        }

        imageDisplayedCount++

        val buffer = image?.planes?.get(0)?.buffer
        val bytes = ByteArray(buffer?.capacity()!!)
        buffer.rewind()
        buffer.get(bytes)
        SaveImageTask().execute(bytes)
    }

    image?.close()
}

As an afterthought, I don't like this wait() in the camera callback thread, and I don't think it is really needed. But most likely, it doesn't really matter, because the actual acquisition happens out-of-process, and the paused ImageReader does not effect the burst.

---

Update Looking at the log that you gathered from the onImageAvailable() callback:

Did you set maxImages to 3? I expected the displayed counter and consumed counter to become different…

The delays between image timestamps are not unifrom, rounded to milliseconds:

33 33 433 200 100 67 167 100 100

which shows that first three frames came without wait for UI to get updated. You could drop the two first frames based on this criterion alone.

As for onCaptureStarted(), the docs are a bot vague (could be on purpose).

This method is called when the camera device has started capturing the output image for the request, at the beginning of image exposure, or when the camera device has started processing an input image for a reprocess request.

Is this or saying that the method may be called 3 times for one capture session? I don't know.

What is explicitly documented, though, is that the timestamps of images are aligned with the timestamp received in onCaptureStarted(). This provides another criterion to filter out the images that were gathered too early.

Actually, I would suggest to log same timestamps from call capture method every time I capture new image in my OnImageAvailableListener listener. It is very possible that this will let your application to capture images in a more predictable way, but in same time.

If you look at the timestamps of the log messages (as reported by logcat, in milliseconds; they may not be aligned with the image timestamps), this can help you understand the timeline even better.