A way to warp an image based on a map

Question

I'm looking for a method where, given an two images (A and B, with A being an RGB image and B being a greyscale image storing the warping information) I can warp A relative to B. To give you some context, this is pretty much the same technique used to artificially create a stereo image from a single image given depth information. In this case A would be the source image and B would be the depth map.

I was able to achieve something similar using this code

def map_pixels(input_image, input_map, N):
    """
    Shifts pixels of the input_image to the right based on the map information and a shift factor N.

    :param input_image: A (3, h, w) numpy array representing the RGB image.
    :param input_map: A (h, w) numpy array representing mapping information, values from 0 (far) to 1 (near).
    :param N: An integer representing the base number of pixels for the shift.
    :return: A (3, h, w) numpy array representing the shifted RGB image.
    """
    
    input_image = np.transpose(input_image, (2, 0, 1))
    input_map = input_map.astype(np.float32) / 255.0
    map_shifts = np.round(N * input_map).astype(int)  # Calculate the shift amount for each pixel

    
    
    # Initialize an array to hold the shifted image
    shifted_image = np.zeros_like(input_image)
    # Iterate through each channel of the image
    for c in range(input_image.shape[0]):  # For each color channel
        channel = input_image[c, :, :]  # Extract the current color channel

        print('CHANNEL SHAPE ' + str(channel.shape))
        # Iterate through each pixel in the channel
        for y in range(channel.shape[0]):  # For each row
            for x in range(channel.shape[1]):  # For each column
                shift = map_shifts[y, x]  # Determine how many pixels to shift
                if x + shift[1] < channel.shape[1]:  # Check if the shifted position is within bounds
                    shifted_image[c, y, x + shift] = channel[y, x]  # Shift the pixel

    shifted_image = np.transpose(shifted_image, (1, 2, 0))
    return shifted_image

The problem I'm facing here is that there is no filtering whatsoever, the pixels just get moved to their new location leaving some broken areas of black pixels and I would like to be able to add some smoothness to the shift.

I was wondering if anyone has any insight in this that they could share.

Note: This is only the function to shift the pixels to the right, I have 3 more functions for left, up and down that I want to be able to call separately.

Also I'm not invested in strictly using Numpy or ImageIO, any solution involving other systems like OpenCV is more than welcome

Thank you