Problem with matching coordinates of the screen and TextRecognizer

Question

I am trying to imeplement liveText on image, and I have implemented it too. But having problem that the point where I click is not matching with the actual line of text, rather it matches with another line.

I am using below library for textRecognition

implementation 'com.google.android.gms:play-services-vision:20.1.3'

My code is

public class ImageViewActivity2 extends AppCompatActivity {

    private ImageView imageView;
    private Bitmap sampleImage;
    private SparseArray<TextBlock> textBlocks;

    // Variables to track text selection
    private float startX;
    private float startY;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_image_view);

        imageView = findViewById(R.id.img);

        Button ocrButton = findViewById(R.id.ocrButton);

        sampleImage = BitmapFactory.decodeResource(getResources(), R.drawable.he);
        imageView.setImageBitmap(sampleImage);

        ocrButton.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                startTextRecognition();
            }
        });

        // Initialize textBlocks
        textBlocks = new SparseArray<>();

        // Set a touch listener on the ImageView

        imageView.setOnTouchListener(new View.OnTouchListener() {
            @Override
            public boolean onTouch(View view, MotionEvent motionEvent) {

                startX = motionEvent.getX();
                startY = motionEvent.getY() ;

                Log.d("Coordinates", "X "+ startX + " Y " + startY+" height="+view.getHeight());
                highlightLineContainingTouchPoint(startX, startY);
                return true;
            }
        });

    }

    private void startTextRecognition() {
        // Initialize the TextRecognizer
        TextRecognizer textRecognizer = new TextRecognizer.Builder(getApplicationContext()).build();

        if (!textRecognizer.isOperational()) {
            // Handle cases where the text recognizer is not operational
            // This may require downloading additional data or checking device settings
            // You can display an error message or prompt the user to enable it
            return;
        }

        // Create a Frame from the loaded image
        Frame frame = new Frame.Builder().setBitmap(sampleImage).build();

        // Perform text recognition
        textBlocks = textRecognizer.detect(frame);
    }

    private void highlightLineContainingTouchPoint(float x, float y) {
        // Initialize a new Bitmap for highlighting
        Bitmap highlightedImage = sampleImage.copy(sampleImage.getConfig(), true);
        Canvas canvas = new Canvas(highlightedImage);
        Paint paint = new Paint();
        paint.setColor(Color.YELLOW);
        paint.setAlpha(128);

        // Iterate over textBlocks to find the line containing the touch point
        for (int i = 0; i < textBlocks.size(); i++) {
            TextBlock textBlock = textBlocks.valueAt(i);
            for (Text line : textBlock.getComponents()) {
                // Check if the touch point is within the bounding box of the line
                if (x >= line.getBoundingBox().left &&
                        x <= line.getBoundingBox().right &&
                        y >= line.getBoundingBox().top &&
                        y <= line.getBoundingBox().bottom) {
                    float left = line.getBoundingBox().left;
                    float top = line.getBoundingBox().top;
                    float bottom = line.getBoundingBox().bottom;
                    float right = line.getBoundingBox().right;
                    Log.e("Coordinates", "Line top"+top+"Bottom"+bottom);

                    // Highlight the line containing the touch point
                    canvas.drawRect(left, top, right, bottom, paint);
                }
            }
        }

        // Update the ImageView with the highlighted image
        imageView.setImageBitmap(highlightedImage);
    }
}

Actually what happens is when I first tried to directly highlight all the text lines in the image then it worked perfectly and at precise location, but I want to implement live text. So I am trying to chekc that the co-ordinates at where I touch is in the range of Bounding box of any line. If so then I highlight that line considering that, that is a desired line. But actually what happens is when I click on the first line then it highlight it perfectly, but now when I touch on 3rd line then it highlights 2nd. and when 7th then highlights 3rd and so on.

So I am expecting efficient and effective solution to impelement Live Text feature.

Answer 1

In the setOnTouchListener you will get view coordinate. You need to change the view coordinates to Image coordinates..

here is the piece of code you need to update!!

    // Set a touch listener on the ImageView
    imageView.setOnTouchListener(new View.OnTouchListener() {
        @Override
        public boolean onTouch(View view, MotionEvent motionEvent) {
            // Convert the view coordinates to image coordinates
            Matrix inverse = new Matrix();
            imageView.getImageMatrix().invert(inverse);
            float[] touchPoint = {motionEvent.getX(), motionEvent.getY()};
            inverse.mapPoints(touchPoint);

            startX = touchPoint[0];
            startY = touchPoint[1];

            Log.d("Coordinates", "X " + startX + " Y " + startY + " height=" + view.getHeight());
            highlightLineContainingTouchPoint(startX, startY);
            return true;
        }
    });

And here is the Output!

Repo Link - https://github.com/wh173d3v11/LiveTextVisionSample