How to scale, crop, and rotate all at once in Android RenderScript

Question

Is it possible to take a camera image in Y'UV format and using RenderScript:

1. Convert it to RGBA
2. Crop it to a certain region
3. Rotate it if necessary

Answer 1

Yes! I figured out how and thought I would share it with others. RenderScript has a bit of a learning curve, and more simple examples seem to help.

When cropping, you still need to set up an input and output allocation as well as one for the script itself. It might seem strange at first, but the input and output allocations have to be the same size so if you are cropping you need to set up yet another Allocation to write the cropped output. More on that in a second.

#pragma version(1)
#pragma rs java_package_name(com.autofrog.chrispvision)
#pragma rs_fp_relaxed

/*
 * This is mInputAllocation
 */
rs_allocation gInputFrame;

/*
 * This is where we write our cropped image
 */
rs_allocation gOutputFrame;

/*
 * These dimensions define the crop region that we want
 */
uint32_t xStart, yStart;
uint32_t outputWidth, outputHeight;

uchar4 __attribute__((kernel)) yuv2rgbFrames(uchar4 in, uint32_t x, uint32_t y)
{
    uchar Y = rsGetElementAtYuv_uchar_Y(gInputFrame, x, y);
    uchar U = rsGetElementAtYuv_uchar_U(gInputFrame, x, y);
    uchar V = rsGetElementAtYuv_uchar_V(gInputFrame, x, y);

    uchar4 rgba = rsYuvToRGBA_uchar4(Y, U, V);

    /* force the alpha channel to opaque - the conversion doesn't seem to do this */
    rgba.a = 0xFF;

    uint32_t translated_x = x - xStart;
    uint32_t translated_y = y - yStart;

    uint32_t x_rotated = outputWidth - translated_y;
    uint32_t y_rotated = translated_x;

    rsSetElementAt_uchar4(gOutputFrame, rgba, x_rotated, y_rotated);
    return rgba;
}

To set up the allocations:

private fun createAllocations(rs: RenderScript) {

    /*
     * The yuvTypeBuilder is for the input from the camera.  It has to be the
     * same size as the camera (preview) image
     */
    val yuvTypeBuilder = Type.Builder(rs, Element.YUV(rs))
    yuvTypeBuilder.setX(mImageSize.width)
    yuvTypeBuilder.setY(mImageSize.height)
    yuvTypeBuilder.setYuvFormat(ImageFormat.YUV_420_888)
    mInputAllocation = Allocation.createTyped(
        rs, yuvTypeBuilder.create(),
        Allocation.USAGE_IO_INPUT or Allocation.USAGE_SCRIPT)

    /*
     * The RGB type is also the same size as the input image.  Other examples write this as
     * an int but I don't see a reason why you wouldn't be more explicit about it to make
     * the code more readable.
     */
    val rgbType = Type.createXY(rs, Element.RGBA_8888(rs), mImageSize.width, mImageSize.height)

    mScriptAllocation = Allocation.createTyped(
        rs, rgbType,
        Allocation.USAGE_SCRIPT)

    mOutputAllocation = Allocation.createTyped(
        rs, rgbType,
        Allocation.USAGE_IO_OUTPUT or Allocation.USAGE_SCRIPT)

    /*
     * Finally, set up an allocation to which we will write our cropped image.  The
     * dimensions of this one are (wantx,wanty)
     */
    val rgbCroppedType = Type.createXY(rs, Element.RGBA_8888(rs), wantx, wanty)
    mOutputAllocationRGB = Allocation.createTyped(
        rs, rgbCroppedType,
        Allocation.USAGE_SCRIPT)
}

Finally, since you're cropping you need to tell the script what to do before invocation. If the image sizes don't change you can probably optimize this by moving the LaunchOptions and variable settings so they occur just once (rather than every time) but I'm leaving them here for my example to make it clearer.

override fun onBufferAvailable(a: Allocation) {
    // Get the new frame into the input allocation
    mInputAllocation!!.ioReceive()

    // Run processing pass if we should send a frame
    val current = System.currentTimeMillis()
    if (current - mLastProcessed >= mFrameEveryMs) {
        val lo = Script.LaunchOptions()

        /*
         * These coordinates are the portion of the original image that we want to
         * include.  Because we're rotating (in this case) x and y are reversed
         * (but still offset from the actual center of each dimension)
         */

        lo.setX(starty, endy)
        lo.setY(startx, endx)

        mScriptHandle.set_xStart(lo.xStart.toLong())
        mScriptHandle.set_yStart(lo.yStart.toLong())

        mScriptHandle.set_outputWidth(wantx.toLong())
        mScriptHandle.set_outputHeight(wanty.toLong())

        mScriptHandle.forEach_yuv2rgbFrames(mScriptAllocation, mOutputAllocation, lo)

        val output = Bitmap.createBitmap(
            wantx, wanty,
            Bitmap.Config.ARGB_8888
        )

        mOutputAllocationRGB!!.copyTo(output)

        /* Do something with the resulting bitmap */
        listener?.invoke(output)

        mLastProcessed = current
    }
}

All this might seem like a bit much but it's very fast - way faster than doing the rotation on the java/kotlin side, and thanks to RenderScript's ability to run the kernel function over a subset of the image it's less overhead than creating a bitmap then creating a second, cropped one.

For me, all the rotation is necessary because the image seen by the RenderScript was 90 degrees rotated from the camera. I am told this is some kind of peculiarity of having a Samsung phone.

RenderScript was intimidating at first but once you get used to what it's doing it's not so bad. I hope this is helpful to someone.