given an image(maybe name it im1) with unknown camera pose, and its sibling image with known camera pose, how to get the camera pose of im1?

Question

Given a image (maybe name it im1) with unknown camera poses, and another image which has a very similar view with im1 but with known camera poses, how can we get the camera pose of im1?

More specifically, I mean we have a pair of stereo images im1 and im2, but I only know the poses of one camera, how can I get the pose of another camera?

Oh, and actually I have the static videos taken by this pair of stereo camera, but the content of the video is not static (with drastically pulling)

learning based or non-learning based solution would be great!

I used Colmap but the processing failed, I think this is due to the drastic motion of video contents.

Answer 1

[Answer uses OpenCv terms throughout]

You can match well-distributed discrete points, then use solvePnP to recover the unknown pose. For example run goodFeaturesToTrack on one image, then search for matches in the other image using opticalFlowPyrLK, and filter out outlier matches using findEssentialMat specifying a robust estimator. Finally, apply solvePnP to the matches.

Answer 2

Given two images capturing the same scene and the camera intrinsics, we can only determine the camera pose up to a scale ambiguity. That's to say, we can know the relative rotation between the two cameras, but the camera distance will have an arbitrary unit.

Depending on the scene, we can do:

• planar scene (e.g. wall): compute homography transform, and then compute relative pose
• 3D scene: compute fundamental matrix, and then use intrinsic parameter to compute essential matrix, and then compute the relative pose.

Here are some tutorials for each step:

• compute the homography transform matrix by matching feature points https://docs.opencv.org/3.4/d1/de0/tutorial_py_feature_homography.html

• Computing camera pose with homography matrix based on 4 coplanar points

• compute the fundamental matrix by matching feature points. See the step by step opencv tutorial epipolar_geometry and cv.findFundamentalMat.

• Pose from Fundamental matrix and vice versa (fundamental matrix -> essential matrix -> pose)