Image crop based on Line/Corner detection with Python

Question

I'm trying to automatically crop all the images from a dataset I have with Python. The problem is that the region of interest in each image is not fixed, so I can't use coordinates for cropping. Below is an example of an image from the dataset and an image showing where I want to crop. All the images look like that: lots of orange around, and in the center is the desired region for cropping.

Input Image:

Input Image with lines that need to be found and cropped:

Desired Output Image:

My problem is that I couldn't find any tool or method that could find the lines of this square and crop them, so I gave it a try with Canny Edge Detection and Harris Corner Detector. Canny seemed to be working on most of the images but couldn't crop them all correctly using the same parameters.

The code I used for Canny:

import cv2 
import os
import numpy as np

input_folder = "./dataset/"
output_folder = "./output_dataset/"

if not os.path.exists(output_folder):
    os.makedirs(output_folder)

for filename in os.listdir(input_folder):
    if filename.endswith(".jpg") or filename.endswith(".JPEG"):
        # Read the image
        image_path = os.path.join(input_folder, filename)
        img = cv2.imread(image_path)

        # Setting parameter values 
        t_lower = 10  # Lower Threshold 10
        t_upper = 70  # Upper threshold 70
        
        # Convert the image to grayscale
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
                
        # Apply GaussianBlur to reduce noise and improve contour detection
        #blurred = cv2.blur(img, (3,3))
        blurred = cv2.GaussianBlur(gray, (3, 3), 0)
        
        # Applying the Canny Edge filter 
        edge = cv2.Canny(blurred, t_lower, t_upper) 
        
        # Find contours in the edged image
        # Example of contour area filtering
        contours, _ = cv2.findContours(edge, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        contours = [cnt for cnt in contours if cv2.contourArea(cnt) > 10]

        
        pts=np.argwhere(edge>0)
        y1,x1=pts.min(axis=0)
        y2,x2=pts.max(axis=0)
        cropped=img[y1:y2, x1:x2]
        
        file_name, file_extension = os.path.splitext(filename)

        # Save the cropped image with the original filename and "_cropped" suffix
        output_path = os.path.join(output_folder, f"{file_name}_cropped.jpg")
        cv2.imwrite(output_path, cropped)

Correct Crop with Canny Example

Input Image:

Canny Edge Detection:

Output Image:

Inorrect Crop with Canny Example

Input Image:

Canny Edge Detection:

Output Image:

Also, I thought of using Harris Corner Detector after applying Canny to the images that didn't crop quite well. By using the Harris Corner Detector, I would get the two farthest detections on the Y-axis of the image and crop it based on them. But I wasn't able to do that correctly; either way, it seems that it would crop most of the images incorrectly:

Harris Corner Detector Output from the previous wrong cropped image with Canny:

Also, I tried detecting vertical and horizontal lines with HoughLines with OpenCV, but it didn't work either. Do you have any suggestions for my problem?

Thanks in advance!!

Answer 1

This is definitely not the most general way, but since what you don't want is very orange (and has high values on the red channel), you can filter these pixels out by some threshold:

red = image[:,:,0].copy()
mask_red_high = (red>180)
red[mask_red_high] = 0

After that, I got a very grainy picture, which I dilated:

kernel = np.ones((20, 20), np.uint8)
red = cv2.morphologyEx(red, cv2.MORPH_DILATE, kernel)

Then it's pretty straight-forward:

blurred = cv2.GaussianBlur(red, (5, 5), 0) # bluring...
edges = cv2.Canny(blurred, 50, 150) # edges...
contours, _ = cv2.findContours(red, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # contours...
sorted_contours = sorted(contours, key=cv2.contourArea, reverse=True) # sort by area...
contour = sorted_contours[1].reshape((-1, 1, 2)) # this one is the second contour, since I couldn't filter out the edges of the image...
epsilon = 0.02 * cv2.arcLength(contour, True) # get epsilon
approximated_square = cv2.approxPolyDP(contour, epsilon, True) # get square
contoured_image = image.copy()
cv2.drawContours(contoured_image, [approximated_square], -1, (255, 255, 255), 2) # draw the square as white
cv2.drawContours(contoured_image, sorted_contours[1], -1, (0, 0, 0), 5) # scatter the simple approximation of the contour
x, y, w, h = cv2.boundingRect(approximated_square) # get params for cropping
cropped_image = contoured_image[y-10:y + h + 10, x - 10:x + w + 10] # cropping...

Would tinker a bit more with dynamic thresholds though, maybe think about Otsu or some other method

V2.0:

I added a few improvements, so that the code would work better on other pictures as well. Namely:

im = cv2.cvtColor(cv2.imread("leaf2.jpg"), cv2.COLOR_BGR2RGB)
image = im.copy()
contoured_image = image.copy()

red = image[:,:,0].copy()
mask_red_high = (red>170)
red[mask_red_high] = 0
kernel = np.ones((30, 30), np.uint8)
red = cv2.morphologyEx(red, cv2.MORPH_DILATE, kernel)

blurred = cv2.GaussianBlur(red, (5, 5), 0) # bluring...
edges = cv2.Canny(blurred, 50, 150) # edges...
contours, _ = cv2.findContours(red, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # contours...
filtered_contours = [contour for contour in contours if 45000 < cv2.contourArea(contour) < 55000] # filter contours according to their area
sorted_contours = sorted(filtered_contours, key=cv2.contourArea, reverse=True) # sort by area...
contour = sorted_contours[0].reshape((-1, 1, 2)) # After filter, there should be only one contour
epsilon = 0.02 * cv2.arcLength(contour, True) # get epsilon
approximated_square = cv2.approxPolyDP(contour, epsilon, True) # get square
contoured_image = image.copy() # copy image
x, y, w, h = cv2.boundingRect(approximated_square) # get params for cropping
cv2.rectangle(contoured_image, (x, y), (x + w, y + h), (255, 255, 255), 5) # draw the square as white
cv2.drawContours(contoured_image, contour, -1, (0, 0, 0), 5) # scatter the simple approximation of the contour
cropped_image = contoured_image[y-10:y + h + 10, x - 10:x + w + 10] # cropping...

This worked for all three images.. yet again, not the most robust way.