How to get the z coordinate from the depth frame obtained by the pyKinectV2 BodyGame example

Question

After playing around a while with the pyKinect2 BodyGame example and doing some research i couldn't figure it out how to receive the joint z value by using the x and y coordinate from this joint.

The author mentioned:

You need to get z from depth frame using x and y you got from body_joints_to_depth_space original post on github

So i fetch the x and y coordinates of the joint in the depth space:

depth_points = self._kinect.body_joints_to_depth_space(joints)
x = depth_points[PyKinectV2.JointType_Head].x
y = depth_points[PyKinectV2.JointType_Head].y

joints is an array of pointers ( i don't know it exactly because of the missing library documentation ) which hold the data of the skeleton joints delivered by the used kinect wrapper. But i have still no idea to get the z coordinate.

The function call

if self._kinect.has_new_depth_frame():
self._depth = self._kinect.get_last_depth_frame()

fetches the last depth frame from the device. self._depth is an array of depth values ( not a matrix ) and i tried indexing the related depth value using x and y. ( x = horizontal position, y = vertical position )

The first option was a classic matrix index call:

z = self._depth[x,y]

but as i mentioned before self._depth is an array. So i tried the second option which means indexing a linear array by using the width of the original image matrix.

z = self._depth[ y * self._frame_surface.get_width() + x ]

but i got a index out of bounds exception.

self._frame_surface.get_width() is the width of the image matrix if i interpret it correctly. ( As i mentioned before there is no documentation for the pyKinectV2 lib ).

So, the question still exist. How to obtain the z value for a skeleton joint?

Answer 1

First of all, I want to point out that I'm not an expert on pyKinectV2 Library. But I have a good expertise in Kinect V2 SDK. After I gone though the pyKinectV2 library source, I saw that pyKinectV2 is just a wrapper for Kinect v2 Official SDK, thus my answer should valid for the pyKinectV2 if it has implemented all the SDK features.

joints[PyKinectV2.JointType_Head].position should return a CameraSpacePoint which should contain the x,y,z coordinate of the joint in Camera Space.

joints[PyKinectV2.JointType_Head].position.X
joints[PyKinectV2.JointType_Head].position.Y
joints[PyKinectV2.JointType_Head].position.Z

Theatrically, this Z value should be equal to the depth value of that specific point.

If you want to find the corresponding depth pixel in depth frame, you can use

self._mapper.MapCameraPointToDepthSpace(joints[PyKinectV2.JointType_Head].position) 

Which will return a DepthSpacePoint. This is what Kinect SDK is using and what pyKinectV2 is using internally.

Your assumption is correct, the depth frame is a 1D 512 x 424 array. First, check the value of self._frame_surface.get_width(), It should return 512. I tried to search _frame_surface object in the pyKinectV2 source, but I couldn't find it, So please validate it first, since Kinect v2 output different frame data, there might be a chance that _frame_surface refers to a another frame who's resolution is different than depth frame,E.g. color frame has 1920 x 1080 resolution.

Finally it is always better to validate the index before accessing it from the array as follows,

if (y * 512 + x) =< 512*424:
   z = self._depth[ y * 512 + x ]

Keep one thing in mind, There is no any guaranty that there is always a corresponding depth pixel and its depth value is the depth of the joint. Because the joints (Skeleton) frame is obtained after BodyIndex frame going though a complex algorithm.

Answer 2

   joint_points_depth = self._kinect.body_joints_to_depth_space(joints)
   x= self.round_int(int(joint_points_depth[PyKinectV2.JointType_SpineMid].x))
   y= self.round_int(int(joint_points_depth[PyKinectV2.JointType_SpineMid].y))
   z = int(self._depth[ y * 512 + x ] ) ## Its in Millimeters

z will give you the distance between you and the camera