Issues while warping image using Homography mat constructed using matches given by BF Matcher

Question

We are trying continuosly process the image frames captured by the two cameras, process every two frames and then stitch them to get a complete view. In order to do so we have 1.extracted surf features. 2.Got the matches between the two images using Flann Matcher. 3.Computed the homography matrix using these matches. 4.Applied warpPerspective on the right image.

//To get the surf keypoints and descriptors:

cuda::SURD_CUDA surf(700);
surf(leftImgGpu, cuda::GpuMat(), keyPointsGpuA, descriptorsAGpu);
surf(rightImgGpu, cuda::GpuMat(), keyPointsGpuB, descriptorsBGpu);
surf.downloadKeypoints(keypointsAGpu, keypoiintsA);
surf.downloadKeypoints(keypointsBGpu, keypoiintsB);

//Flann based matcher:
FlannBasedMatcher matcher(new cv::flann::KDTreeIndexParams(4), new 
cv::flann::SearchParams())

//To get the homography matrix:
vector<Point2f> imgPtsA, imgPtsB;
for(int i=0;i<matches.size();i++){
    imgPtsB.push_back(keypointsB[matches[i].queryIdx].pt);
    imgPtsA.push_back(keypointsA[matches[i].trainIdx].pt);
}
Mat H=findHomography(imgPtsA, imgPtsB, CV_RANSAC);
 
//To a warp right image:
warpPerspective(rightImg, warpRight, H, rightImg.size());

We have two issues: Issue 1: The warped image3 is moving around. The left and right cameras are fixed and the images( left, right) we are processing is almost the same everytime. We suspect there is some issue with the matches and the homography matrix becuase of which the warped image is not coming properly. Issue 2: We intially used BF Matcher to get the matches. When constructed the Homograpy mat using these matches, we were getting weird results. After using the Flann based matcher the result was comparatively better.

Answer 1

To create a proper "panorama" image with stitching the cameras need to be on nearly on the same position in space, otherwise parallax errors will occur (see here). In general case, a homography can only warp a single plane within the image such it is registered with its pendant. Thus, it would be possible e.g. to stitch just the floor (if it provides enough texture for features of course).

So, you can not expect a stable result, since the homography can not model this transformation. More vividly: The front side of the chair is only visible in the right image, so it is not possible to "match" this area with the left image.