OpenCV Camera calibration and SolvePNP translate results

Question

I am attempting to initially calibrate a sensor using a chessboard. I make around 50 run, and after I calibrate the camera I proceed in using solvepnp to teach the coordinate system, and since I am using a well defined chessboard to actually learn the real world coordinates.

As input for the SolvePnP I use all corner values with their corresponding real world distances.

My biggest issue is that the translate matrix I calculate from the SolvePvP is a bit strange. It is my understanding that the translate matrix is the actual offset between the camera and the coordinate system, which I define as the the upper part of the chessboard. But I get completely random values, with the Tz gives me a value of around 6000, even though the distance between camera and chessboard is around 1600 mm, and I am not even using depth values as parameters for the solvePnP method anyway.

Any input on what could be off?

Code Sample:

50 x DrawChessboardCorners
Corner result: {X = 1170.45984 Y = 793.002}
               {X = 1127.80371 Y = 792.54425}

3d Points:     {X = 175 Y = 70, Z = 0}
               {X = 140 Y = 70, Z = 0}

total 18 (6x3) results per run for a total of 50 runs.

Afterwards I calibrate the camera:

CalibrateCamera(_objectsList, _pointsList,
                new Size(_sensorWrapper.Width, _sensorWrapper.Height),
                cameraMatrix, distCoeffs,
                CalibType.Default,
                new MCvTermCriteria(30, 0.1), out _, out _);

Afterwards, using the cameraMatrix and distCoeffs I use SolverPnP by using the top left, top right, bottom left bottom right corners, with their real world coordinates.

The results i get from the calibration are :

    {
  "CameraMatrix": [
    [
      5969.947,
      0.0,
      959.687256
    ],
    [
      0.0,
      6809.737,
      540.3694
    ],
    [
      0.0,
      0.0,
      1.0
    ]
  ],
  "DistortionCoefficientsMatrix": [
    [
      0.141516522,
      285.377747,
      0.008248664,
      0.0280253552,
      1.5376302
    ]
  ],
  "RotationMatrix": [
    [
      0.9992069,
      -0.0270648878,
      0.0292078461
    ],
    [
      -0.0003847139,
      0.726907134,
      0.68673563
    ],
    [
      -0.0398178138,
      -0.6862022,
      0.7263202
    ]
  ],
  "TranslationMatrix": [
    [
      22.5370159
    ],
    [
      -362.535675
    ],
    [
      5448.58057
    ]
  ],
  "SensorName": "BRIO 4K Stream Edition",
  "SensorIndex": 0,
  "Error": 0.18790249992423425
}