How to detect and crop individual sprites when their bounding boxes overlap?

Question

Please refer to this link first:

How to automatically detect and crop individual sprite bounds in sprite sheet?

In the example in the above link, the bounding boxes of the individual sprites do not overlap, so the code in the accepted answer works well, and all the sprites are extracted in individual files.

But supposed the sprites were packed differently such that their bounding boxes overlapped even thought the individual images did not overlap:

Then the above code would not work well at all, and just one file is output because the bounding boxes would intersect.

How would someone solve this problem using OpenCV?

Without using OpenCV, my approach would be to:

• Iterate through each sprite till I found a non-transparent sprite. Store it, and all connected non-transparent sprites as an individual set.
• Once no more connected pixels from the previous set, examine the remaining pixels in the rest of the image, ignoring the ones I have already seen and put in the above set. When I encounter a new non-transparent pixel I have not seen before, then repeat the same as the first step, but place these connected pixels in a new set.
• Like that repeat the process till there are no pixels left to examine.
• For each set of connected pixels, create a new image with the dimensions from a bounding box constructed by the farthermost pixels of that set. But include only the pixels from that connected set in the output image. Repeat for each set to create a new image for each connected set.

Instead of the approach I have described above, is there a method to use OpenCV to do this ? I can do it using my method, but I would like to learn more about OpenCV and what it is capable of, but I am just a beginner and still learning.

Answer 1

You can fill each contour with white and then generate an image with only that contour. After that, apply a regular cropping by getting the properties of the bounding rect. Here is the code:

im = cv2.imread("trees.png") # read image
imGray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY) # convert to gray
contours, _ = cv2.findContours(imGray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # contouring
sortedContours = sorted(contours, key=cv2.contourArea, reverse=True) # sorting, not necessary...
for contourIdx in range(0,len(sortedContours)-1): # loop with index for easier saving
    contouredImage = im.copy() # copy image
    contouredImage = cv2.drawContours(contouredImage, sortedContours, contourIdx, (255,255,255), -1) # fill contour with white
    extractedImage = cv2.inRange(contouredImage, (254,254,254), (255,255,255)) # extract white from image
    resultImage = cv2.bitwise_and(im, im, mask=extractedImage) # AND operator to get only one filled contour at a time
    x, y, w, h = cv2.boundingRect(sortedContours[contourIdx]) # get bounding box
    croppedImage = resultImage[y:y + h, x:x + w] # crop
    cv2.imwrite("contour_"+str(contourIdx)+".png", croppedImage) # save

And the results look nice as well, I'll just post the biggest one and smallest one as an example:

V2.0: make it work with both images

The first image, for some reason, has 255 on the green channel for all the background. Which is why only one big contour is detected. I added a few lines of code to pre-process the image and also to take into consideration the alpha channel. Here is the code:

im = cv2.imread("trees.png", cv2.IMREAD_UNCHANGED) # read image
b, g, r, alpha = cv2.split(im.copy()) # split image
g[g==255] = 0 # for the first image where the green channel has 255 on all background pixels
imBGR = cv2.merge([b,g,r]) # recombine image in BGR format
imGray = cv2.cvtColor(imBGR, cv2.COLOR_BGR2GRAY) # convert to gray
contours, _ = cv2.findContours(imGray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # contouring
sortedContours = sorted(contours, key=cv2.contourArea, reverse=True) # sorting, not necessary...
for contourIdx in range(0,len(sortedContours)-1): # loop with index for easier saving
    contouredImage = imBGR.copy() # copy BGR image
    contouredImage = cv2.drawContours(contouredImage, sortedContours, contourIdx, (255,255,255), -1) # fill contour with white
    extractedImage = cv2.inRange(contouredImage, (254,254,254), (255,255,255)) # extract white from image
    resultImage = cv2.bitwise_and(imBGR, imBGR, mask=extractedImage) # AND operator to get only one filled contour at a time
    x, y, w, h = cv2.boundingRect(sortedContours[contourIdx]) # get bounding box
    croppedImage = resultImage[y:y + h, x:x + w] # crop
    cv2.imwrite("trees_2_contour_"+str(contourIdx)+".png", croppedImage) # save

I won't bother with adding images of the results again, as they are the same as before. However, here is an image of the first example with the green background I am talking about:

The code:

im = cv2.imread("trees2.png", cv2.IMREAD_UNCHANGED) # read image
b, g, r, alpha = cv2.split(im.copy()) # split image
imBGR = cv2.merge([b,g,r]) # recombine image in BGR format
plt.imshow(imBGR)

The image:

I went around this with g[g==255] = 0