Body tracking has low precision on Azure Kinect

Question

I am currently working on a project where the Azure Kinect is meant to be used as an interaction method for a 360° Screen room.

When using the Body Tracking Viewer from Microsoft the camera recognizes me well and precisely but in my application the calculated distance (using distanceTo from each vector) between my handtip joint and thumb joint is all over the place. Without moving my hand it will go from 10mm to 120mm. I absolutely need this to be a bit more precise to be able to select things with the hand.

I suppose I have a problem in my startup configuration in the camera which causes it to be less precise than the Body Tracking Viewer but I don't know where to look.

As I wasn't sure if the default tracker config uses the lite dnn_model_2_0 or the "big" one, I tried setting it manually tracker_config.model_path = "dnn_model_2_0_op11.onnx"; but with the same result. Going to near field of view instead of wide does help a little bit but it stays very janky. Especially the left hand is all over the place, the right one is almost precise.

Here the startup/config from my cpp project

[...]
MQTTClient client;
MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;
MQTTClient_message pubmsg = MQTTClient_message_initializer;

int maint()
{
k4a_device_t device = NULL;
    VERIFY(k4a_device_open(0, &device), "\nOpen K4A Device failed! Is the camera connected?");


    // Start camera. Make sure depth camera is enabled.
    k4a_device_configuration_t deviceConfig = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL;
    deviceConfig.depth_mode = K4A_DEPTH_MODE_WFOV_UNBINNED;
    deviceConfig.color_resolution = K4A_COLOR_RESOLUTION_OFF;
    deviceConfig.camera_fps = K4A_FRAMES_PER_SECOND_15;
    VERIFY(k4a_device_start_cameras(device, &deviceConfig), "\nStart K4A cameras failed!");

    k4a_calibration_t sensor_calibration;
    VERIFY(k4a_device_get_calibration(device, deviceConfig.depth_mode, deviceConfig.color_resolution, &sensor_calibration),
        "Get depth camera calibration failed!");

    k4abt_tracker_t tracker = NULL;
    k4abt_tracker_configuration_t tracker_config = K4ABT_TRACKER_CONFIG_DEFAULT;
    VERIFY(k4abt_tracker_create(&sensor_calibration, tracker_config, &tracker), "\nBody tracker initialization failed!");

[...]
}