I discovered the other day that my camera lens or perhaps tilted sensor is causing an object centered in front of the camera to appear off center in the captured image. As close as I can get it, the red circle is the point that should be aligned directly to the camera lens, the orange circle is the center pixel of my image. In a perfect world these would be aligned. This is the original picture, not the undistorted version using the results of my camera calibration. This is using Python and OpenCV just to put that out there.
I used solvePnP() so that I can map the pixel coordinates to real world coordinates and the results are better than I could have expected. The the output in real world coordinates is bang on, the distance it determined is bang on and it shows a shifting z value that is consistent with the rotation of a plane caused by a center shift in this direction.
What I would like to do instead is re-project this image as if the camera was not skewed and taking a picture head on (black pixels around the outside are okay and expected). I don't want to translate the image, but am looking to fix for rotation. I think a re-projection to fix rotation and redo of solvePnP() will get me what I want? Really, the goal is to correct those real world coordinate outputs so that x and y remain accurate, but z will be constant (as you would expect with an orthogonal plane and camera that was "perfect"). I am not sure how to do the reprojection using the output from my first run of solvePnP(), and am hoping someone can confirm that doing another solvePnP() after re-projection would help me get consistent z values. Or perhaps there is a better way to do all of this?
Output based on some important coordinates in my image (outputs are in mm):
Center Pixel (orange circle):
Shift in mm is confirmed as accurately as I can with a ruler.
u:319
v:239
[-12.19730412]
[-13.78697338]
[-0.14210989]
Known True Center (red circle):
A perfect mapping would be 0,0,0 we are pretty close.
u:359
v:195
[0.50044927]
[0.11737794]
[0.03856228]
Top Left Corner:
Coordinates are within 1mm and z is pushed back as I would expect with the tilt displayed in the image.
u:0
v:0
[-112.58447447]
[62.540533]
[-0.43397294]
Bottom Right Corner:
Coordinates are within 1mm and z is pulled forward as I would expect with the tilt displayed in the image.
u:639
v:479
[88.50417125]
[-90.43334375]
[0.15017949]
I have thought of just using the displacement to calculate my two adjustment angles and use these as corrections, but that seems like garbage way to do it when OpenCV clearly has ways of doing this. Also I would not be correcting for any roll if I tried that.
Any help or suggestions would be greatly appreciate!
Thanks all!