ICP doesn't do much for my image registration of hands, why?

Question

I edited the complete question on 14-3-2023

---

I am trying to align two images of hands because I want to align one image to a reference image.

For this, I would love to use ICP, but up until now, all my ICP matrixes come back empty.

The preprocessing of the photo's is first to make a mask to seperate the hands from the background.

After that I try to already align the images the best I can using the bounding box technique and to reshape the size to the reference hand after that.

That results into these two pictures:

When using my ICP, the matrix returns empty (see attachment for ICP code). Eventually I expect that the two contours morph as closely to one line of a hand as possible.

What is happening?

I also tried to match the images using the ORB detection algorithm, but then it matches complety wrong points.

Questions are:

1. Why does my ICP return empty? (Also the filled in hand instead of a contour returns an empty matrix)
2. Is there another technique I could use rather than ICP or ORB?

Please help :)

ICP:

import cv2
import numpy as np
from scipy.spatial import cKDTree

def icp(left_hand, right_hand, num_iterations=20, threshold=0.001):

# Detect keypoints and descriptors
sift = cv2.SIFT_create()
kp1, desc1 = sift.detectAndCompute(left_hand, None)
kp2, desc2 = sift.detectAndCompute(right_hand, None)

# Match keypoints using brute force matching
#bf = cv2.BFMatcher()
#matches = bf.match(desc1, desc2)

#Match keypoints using FLANN matching
flann = cv2.FlannBasedMatcher()
matches = flann.match(desc1, desc2)

# Get corresponding points
left_pts = np.float32([kp1[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
right_pts = np.float32([kp2[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

# Initialize transformation matrix
M = np.identity(3)

# Iterative Closest Point algorithm
for i in range(num_iterations):
    # Find closest points using KDTree
    tree = cKDTree(right_pts.reshape(-1, 2))
    dist, indices = tree.query(left_pts.reshape(-1, 2))

    # Compute transformation matrix using Singular Value Decomposition (SVD)
    A = np.zeros((left_pts.shape[0], 6))
    b = np.zeros((left_pts.shape[0], 1))
    for j in range(left_pts.shape[0]-1):
        x, y = right_pts[indices[j]][0]
        A[j] = np.array([left_pts[j][0][0], left_pts[j][0][1], 0, 0, 1, 0])
        A[j+1] = np.array([0, 0, left_pts[j][0][0], left_pts[j][0][1], 0, 1])
        b[j] = x - left_pts[j][0][0]
        b[j+1] = y - left_pts[j][0][1]

    x, residuals, rank, s = np.linalg.lstsq(A, b, rcond=None)
    delta_M = np.vstack((x.reshape(2, 3), np.array([0, 0, 1])))
    
    #Check if the algorithm has converged
    if np.linalg.norm(delta_M - np.identity(3)) < threshold:
        break

    # Update transformation matrix
    M = delta_M.dot(M)
    
    # Transform left points using updated transformation matrix
    left_pts = cv2.transform(left_pts, M)

# Warp left hand image to align with right hand image
aligned = cv2.warpPerspective(left_hand, M, (right_hand.shape[1], right_hand.shape[0]))

return aligned

# Warp left hand image to align with right hand image
aligned = cv2.warpPerspective(left_hand, M, (right_hand.shape[1], right_hand.shape[0]))

return aligned

Answer 1

Sorry for taking time to get back to you, and that this will not be an answer you were looking for. I just had lots to comment on which will not fit in the comment box.

First off, your code sample got mangled by the autoformatter. Try editing it and fencing off your entire code block in triple backticks like this: ```code goes here```. It is currently too hard to make out the indentation levels, which is very important in python.

---

However, the provided info isn't what I was looking for either.

What I wanted to know was what you expected to happen after you ran ICP on your images. Do you expect the hands to just move around in positions? rotate? or morph in some other ways? From what I see, your hand images had changed between the first and second image; cropped differently and swapped order which you've plotted. Is this expected behavior? Because maybe the code already did work, just not as you've expected. (Again, what are you expecting to happen from the ICP algorithm?)

As for the code snippet, I was also looking for the code you've used to plot the images shown. A common mistake is you are plotting the incorrect variables, hence no change is shown. Currently I cannot tell, because I can't find anything wrong with the provided code.

So distilling your code down to the bare minimum example will really help here. Just create a toy example. Two toy example images you want to perform image registration on. A line of ICP. Another line to plot results. Etc.

---

Finally, if you are looking to "merge" two images in python, there are many approaches and plenty more tutorials on how to "overlay" two images on top of one another.
Assuming you want to overlay variables created from cvright and cvleft, I think those resulting variables will also be 2D arrays of binary digits. So the simplest approach I can think of is to use matplotlib's pcolor() function and do something like:

import matplotlib.pyplot as plt

# Your code to load / define your images go here

plt.pcolor(cvright, alpha=0.5)
plt.pcolor(cvleft, alpha=0.5)
plt.show()

Test something like this out first, preferrably on examples that you know what the final results should look like. Tinker around with the code until you can get it to output what you expect it to, then try it with your actual hand images.

Hope this atleast helps guide your question somewhat.