Stereo Reconstruction Problems using BM and SGBM approaches

Question

I've calibrated my stereo camera using OpenCV's chessboard calibration routine. I am currently using the obtained intrinsic and extrinsic matrices to rectify my messages and use a block matching approach (I've used both BM and SGBM and get the same results) to eventually go to a 3D point cloud.

Unfortunately, I am finding problems in my 3D point cloud and for some reason, there seems to be an errorenous "step" in terms of depth values.

I am used a simple experiment of imaging a pile of dirt that has a lot of texture, I expected to see a consistent reconstruction, but what I find is that a portion of it, is at the right depth value, whereas the rest of it seems to be flat on the ground. I've tried varying my stereo mark size, the max/min disparity values etc. But still I don't see any change. I've also tried imaging it horizontally AND vertically, but the problem remains in both views.

Can anyone please suggest what I might be doing incorrectly?

For the moment, the simpler BM parameters are mentioned below:

int ndisparities = 16*15;//16*10;  
int SADWindowSize = 31; 
cv::StereoBM sbm( CV_STEREO_BM_BASIC, ndisparities, SADWindowSize );    
sbm( rectified_imgRight, rectified_imgLeft, imgDisparity16S, CV_16S );
imgDisparity16S.convertTo(imgDisparity8U, CV_8U, 255/(ndisparities*16.));
cv::imwrite("disparity.png", imgDisparity8U);

Answer 1

Your disparity map doesn't look quite right to me. Except for the black holes, everything else has the same intensity in the disparity map. This cannot be right.

The general rule is this: lower the depth, higher is the disparity.

I suggest you check your disparity map again.

Also, once you obtain a proper disparity map, you should do a hole-filling to take care of all the black areas. This will definitely affect your reconstruction.

Answer 2

Your disparity results look great, I see. No problem there.

What is your camera setup, a normal horizontal stereo setup? (by the looks of the shoe in the image this might be the case)

Also, it seems that the 3d mound is to the left of the pictured item: have you checked that you are overlaying your 3d data with the correct camera image? By default, OpenCV algorithms will give a left rectified image depth map, but in your case, looks like the object image has shifted to the right. In my experience this is because a depth map is compared to the right rectified image, which is wrong.

Have you checked that you are trying to compare the depth map against the correct image (left camera rectified image)? Also check the orientation of the image.

You will get the correct height in no time, I believe.