How to get Subpixel Disparities in AVDepthData Objects

374 Views Asked by At

I'm working on a swift app that should measure physical correct z-values of a person standing in front of a statically mounted iPhone 7+. Therefore, I am using AVDepthData objects that contain the depth maps coming from the Dual Camera System.

However, the resulting point clouds indicate that the depth maps do not have subpixel accuracy since the point clouds consist of slices along the z-direction and the distances of adjacent slices increase with increasing depth. This seems to be caused by the integer discretization.

Here are two files that visualize the problem:

Captured Depthmap, cropped Z-Values after 4.0m: DepthMap with Z-values in Legend

Textured Pointcloud, view from the side (90°): Pointcloud rendered from iPhone

According to Apple's documentation, I've already deactivated the temporal filtering and unwarped the images using the distortion coefficients from the lookup table, in order to have correct world coordinates.

Filtering depth data makes it more useful for applying visual effects to a companion image, but alters the data such that it may no longer be suitable for computer vision tasks. (In an unfiltered depth map, missing values are represented as NaN.)

Is there any way to retrieve depth maps that have subpixel accuracy in order to perform good measurements of a person standing in front of the camera?

Below you can find the python code I wrote to create the pointclouds offline, the method calculate_rectified_point was provided by Apple to remove lense distortion from the images.

for v in range(height):
    for u in range(width):
        r, g, b = rgb_texture[v, u]
        z = depth_map[v, u]

        if z <= 0:
            continue

        # Step 1: inverse the intrinsic parameters
        x = (u - center[0]) / focal_lengths[0]
        y = (v - center[1]) / focal_lengths[1]

        # Step 2: remove the radial and tangential distortion
        x_un, y_un = calculate_rectified_point((x, y), dist_coefficients, optical_center, (width, height))

        # Step 3: inverse extrinsic parameters
        x, y, z = extrinsic_matrix_inv.dot(np.array([x_un * z, y_un * z, z]))
0

There are 0 best solutions below