Superimposing 2D heat map on 3D Image with transparency

Question

I have a 3D python Image and its 2D feature vector . I displayed each with imshow() as shown in the code down here and I can see it clearly. What I wanted now is to overlay the 2D feature vector as a heat map on top of its 3D image. I tried adding them but dimension issue was raised , I expanded the 2D feature vector by adding third dimension but superimposed image is messed up. The dimension are are here :- feature >> (32,96) , image >> (32,96,3)

img_feature = np.uint8(feature22 + img_raw_np_resize)

ax0.imshow(img,extent =extent)     
ax1.imshow(feature,alpha = 0.75, interpolation = 'gaussian', cmap = plt.cm.jet,extent =extent)  
ax2.imshow(img_feature,alpha = 0.75, interpolation = 'gaussian', cmap = plt.cm.jet,extent =extent)

extent refers to make all of them display in same size and its is defined already. here is what I go from my attempt by the above code

The display I am looking for something like below ...

and here is the pesudo-code that does this ( I am trying to produce minimal code to share but it is taking time - so I share this in case it helps to figure out the issue : -

def save_feature_to_img_2(self):

    feature = (
        get_feature()
    )  #  get feature of size -- > torch.Size([1, 256, 32,96])

    feature2 = features[
        :, 0, :, :
    ]  #  I pick one from the 256 feature maps - [1,1,32,96]

    feature2 = (
        feature2.data.numpy()
    )  # convert it to numpy array for later fusion with image array - [1,1,32,96]

    features2 = features2.view(
        features2.shape[1], features2.shape[2]
    )  # reshape into 2D feature map  [32,96]

    img_raw1 = Image.open(self.img_path)  # read raw image size [128,64,3]

    img_raw_np = np.array(img_raw1)  # imge to numpy array

    newsize = (h, w)  # shape to re-size image to same size a feature
    img_raw_resize = img_raw1.resize(newsize)
    img_raw_np_resize = np.array(img_raw_resize)  # size is now [32,96,3]

    # use sigmod to  normalize feature to  [0,1]
    feature2 = 1.0 / (1 + np.exp(-1 * feature2))

    # display setup
    dx, dy = 0.05, 0.05

    y = np.arange(-8, 8, dx)  # Y - axis range
    x = np.arange(-4.0, 4.0, dy)  # X -axis range
    X, Y = np.meshgrid(x, y)  # meshgrid to enclose my display into
    extent = (
        np.min(x),
        np.max(x),
        np.min(y),
        np.max(y),
    )  # extent - the X and Y range  - just to make unifrom display

    feature2 = (255.5 * feature2 / np.amax(feature2)).astype(
        np.uint8
    )  # put feature into [0-255] range for colour image

    # to display img, feature and img_feature
    fig = plt.figure()
    ax0 = fig.add_subplot(131, title="Image")
    ax1 = fig.add_subplot(132, title="Heatmap")
    ax2 = fig.add_subplot(133, title="overlayed")

    # hereunder I used this code share in the answer to fuse
    alpha = 0.5

    img_heatmap = (
        feature2[:, :, None].astype(np.float64) * alpha
        + img_raw_np_resize * (1 - alpha)
    ).astype(np.uint8)

    ax0.imshow(img_raw1, alpha=1.0, interpolation="gaussian", cmap=plt.cm.jet)
    ax1.imshow(feature2, alpha=1.0, interpolation="gaussian", cmap=plt.cm.jet)
    ax2.imshow(
        img_heatmap, alpha=0.7, interpolation="gaussian", cmap=plt.cm.jet
    )

    cv2.imwrite("./img_heatmap.jpg", img_heatmap)

This is the new display I got

I use the following to fuse image and feature ...

alpha = 0.5         
img_feature = ((plt.cm.jet(feature2)[:, :, :3] * 255) * alpha + (1-alpha)*img_raw_np_resize).astype(np.uint8)  and displaying it with ax2.imshow(img_feature,alpha = 0.7, interpolation = 'gaussian', cmap = plt.cm.jet)

Answer 1

Variant 1

If you want to show only heated region of image you have to multiply by heatmap instead of adding.

Formula for you will be img_feature = (img * (feature[:, :, None].astype(np.float64) / np.amax(feature))).astype(np.uint8).

Full example code (with my own image and auto-generated example heatmap):

Try it online!

import requests, PIL.Image, io, numpy as np, matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h, w, _ = img.shape
mg = np.mgrid[:h, :w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
img_feature = (img * (feature[:, :, None].astype(np.float64) / np.amax(feature))).astype(np.uint8)
# show images
fig, (ax0, ax1, ax2) = plt.subplots(1, 3)
ax0.imshow(img)
ax1.imshow(feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
ax2.imshow(img_feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
plt.show()

Output:

---

Variant 2

If you want just brighten (whiten) regions that are heated and darken (blacken) regions that are un-heated you just do alpha blending by formula alpha = 0.5; img_feature = (feature[:, :, None].astype(np.float64) * alpha + img * (1 - alpha)).astype(np.uint8).

Try it online!

import requests, PIL.Image, io, numpy as np, matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h, w, _ = img.shape
mg = np.mgrid[:h, :w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
alpha = 0.5; img_feature = (feature[:, :, None].astype(np.float64) * alpha + img * (1 - alpha)).astype(np.uint8)
# show images
fig, (ax0, ax1, ax2) = plt.subplots(1, 3)
ax0.imshow(img)
ax1.imshow(feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
ax2.imshow(img_feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
plt.show()

Output:

---

Variant 3

Same as Variant-2 (with alpha-blending) but instead of black-and-white feature array is used RGB feature array based on matplotlib.pyplot.cm.jet coloring scheme.

As you can see from code you can use any coloring scheme in expression plt.cm.jet(feature) instead of plt.cm.jet colors.

Try it online!

import requests, PIL.Image, io, numpy as np, matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h, w, _ = img.shape
mg = np.mgrid[:h, :w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
alpha = 0.5; img_feature = ((plt.cm.jet(feature)[:, :, :3] * 255) * alpha + img * (1 - alpha)).astype(np.uint8)
# show images
fig, axs = plt.subplots(2, 2)
axs[0, 0].imshow(img)
axs[0, 1].imshow(feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
axs[1, 0].imshow(img_feature, alpha = 1., interpolation = 'gaussian', cmap = plt.cm.jet)
axs[1, 1].remove()
plt.show()

Output:

---

PS. I just noticed that Matplotlib does normalization (feature - min) / (max - min) when drawing heatmap hence I decided to do same thing in overlaying formula, final formula becomes such:

alpha = 0.5; img_feature = ((
    plt.cm.jet(
        (feature - np.amin(feature)).astype(np.float32)
        / (np.amax(feature) - np.amin(feature)).astype(np.float32)
    )[:, :, :3] * 255
) * alpha + img * (1 - alpha)).astype(np.uint8)

Example code of using formula above is here (or here, and for your images). Resulting image of using formula above is here (and for your images). Final your image overlayed: