My current task is to render faces of a 3D mesh, and render the edges as thick lines (of variable thickness). But I want to perform hidden line removal for the lines obscured by the faces.
In the past I simply implemented a polygon offset when rendering faces, and it was sufficient. I was not getting any z-fighting. However, now I'm polishing rendering routines and I need to draw lines of variable thickness. I currently have my lines going through a geometry shader that turns each line in to a very thin quad whose is variable. However, polygon offset is no longer effective especially at higher resolutions. I realized I simply just have to cull any line primitive that is hidden by a face, and then draw the rest of the lines with no depth test so every fragment is drawn entirely on top of the faces.
My question is how can I perform culling like this? But I would want to do this on the GPU, because performing intersection tests for each edge vertex and each face on each frame is crazy expense to do on the CPU.
--- MY SOLUTION ATTEMPT ---
My sort of bootstrap method of culling lines on GPU is:
- Draw the faces already with a constant polygon offset into the framebuffer and store z-values.
- Draw the lines and test whether not to cull the line in the geometry shader: Sample the z-buffer at the screen coordinates of each line-vertex (using GL_NEAREST) to test if the vertex is behind the face or not. Then emit all quad vertices only when both endpoints of the line are not obscured.
--- ANALYSIS ---
This seems like it might be better than CPU culling obviously, but I don't know if this is the best way to do it. Having an if-statement in the geometry shader doesn't seem too smart to me. But then again, it shouldn't have much worse performance, because I think for the most part, GPUs perform both branches of an if-statement to maintain SIMD. But if one of the branches of the if-statement simply does nothing and ends primitive, then I don't think it will have much of a noticeable impact at all.
As for edge cases, I can't think of any immediate cases that would cause an unwanted result (aside from polygon offset problems or two faces being extremely close to each other. But that exists in polygon offset methods regardless). It ultimately works because we will eventually quantize the endpoints of the lines at the pixel stage anyways. Sampling the z-buffer at nearest is (seemingly?) equivalent to that.
Is this optimal? Is there a better/cleaner way to do this? Is there a form of line culling built into OpenGL?