Based on the results I got and the provided documentation I concluded that calibration using non-planar rig does not work in OpenCV (they are heavily dependent on the initial guess). According to their documentation of the function calibrateCamera
The algorithm performs the following steps:
- Compute the initial intrinsic parameters (the option only available for planar calibration patterns) or read them from the input parameters. The distortion coefficients are all set to zeros initially unless some of CV_CALIB_FIX_K? are specified.
- Estimate the initial camera pose as if the intrinsic parameters have been already known. This is done using solvePnP
- Run the global Levenberg-Marquardt optimization algorithm to minimize the reprojection error, that is, the total sum of squared distances between the observed feature points imagePoints and the projected (using the current estimates for camera parameters and the poses) object points objectPoints. See projectPoints for details.
Also in mandatory you need to provide initial estimation of intrinsic if you use non-planar rig as an input, thereby confirming the first point.
So, my question is, is there any any library (any language) available which has the implementation of finding intrinsics (or maybe together with the extrinsics) for non-planar rig?
If you have access to Matlab, or can understand Matlab syntax and translate it in your favorite one (easy from Matlab to Python and you get openCV integration). You can take a look at their non planar rigs. This code is based on the works of Heikkilä:
"Geometric Camera Calibration Using Circular Control Points", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 10, pp. 1066-1077, Oct 2000.
And including an example of such calibration, these links contains the corresponding code:
http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example3.html
http://www.ee.oulu.fi/~jth/calibr/
I hope this broad answer is useful.