How to get the z coordinate (distance) from a Kinect v2 RGB coordinate?

Question

I am trying to convert libfreenect RGB coordinates into DEPTH coordinates, in order to find the z coordinate (distance of object). I tried so many things but I was not able to get the resolutions right. Any ideas?

I am using rgbmat in order to apply Kalman filter. So I only have the RGB center coordinates of an object:

libfreenect2::Frame undistorted(512, 424, 4), registered(512, 424, 4), depth2rgb(1920, 1080 + 2, 4);
libfreenect2::Frame *rgb = frames[libfreenect2::Frame::Color];
Mat(rgb->height, rgb->width, CV_8UC4, rgb->data).copyTo(rgbmat);
registration->apply(rgb, depth, &undistorted, &registered, true, &depth2rgb);

Finding xyz coordinates from undistorted frame, with input centerX and centerY from rgb frame:

float centerX, centerY, centerZ;
registration->getPointXYZ(&undistorted, center.y, center.x, centerX, centerY, centerZ); 

With this code the output is always wrong. Most of the time "inf" or "-inf" values. When I perform my tracking algorithm on registered frames, the output (distance) is correct, although I cannot use registered frames because I need to analyse RGB frames.

I also tried changing "undistorted" in getPointXYZ() to "depth2rgb", but no success :(

Is there a way to get RGB values out of registered frames?