Obstructed/Incomplete quadrilateral detection with OpenCV

Question

I want to detect quadrilateral with OpenCV. Shape might not be in diffrent colour than surface. e.g, It could be a rectangle hole on the surface.

My current approach works great when:

• The object is on a plain surface with high color contrast on the edges.
• Edges are not obstructed.

Here is my current pipeline:

• Convert to Gray Scale
• Median Blur (GaussianBlur removes edge details)
• Canny edge detection with OTSU threshold values
• dilate & erode to fill small gaps
• Contour detection
• Filter out contours by min area, max points & roundness of shape
• approxPolyDP to get 4 points

Problem with this approach:

• When there is not much color contrast on the edges.
• There is a light reflection on the edge, it misses that portion of the edge
• When the object is in touch with another object or corner of the surface.

Here is results for different scenarios:

• Work great when background is clearly seprated out
• Edges are incomplete because of light reflaction or less color contrast
• Edges are obstruced by something
• Edges are conneted to other object and not seprately visible

Note: Here i have used an book as example. It could be any quadrilateral item even surface hole.

I have also tried with HoughLine approach but it results in too many lines and makes it impossible to filter lines related to rectangle. Followed blog from DropBox but can't figur out how to filter rectanlge lines.

Please let me know if there is any better approach that solves these problems. Any lead will be helpful.