Here's my situation:
I'm using OpenGLES 3.0.
I'm doing z-buffer shadow casting. HOWEVER, I also have some cuts in the scene that are done with a stencil buffer, i.e. to dig "holes" in the ground that are not geometry.
I am using a stencil buffer to not allow shadowing in the holes. BUT, the problem with this is, the stencil buffer prevents shadowing all the way forward in the scene, so if something occludes the hole but is also in shadow, I will see a nonshadowed "hole" where there should be shadow.
So, my plan was to have a second z-buffer, and simply write the stencil masks into that zbuffer. When trying to draw my shadow pass, I would simply check my z-value against the other zbuffer and skip drawing if greater than or equal, but allow drawing if less than.
However, I don't seem to be able to read the value of the other z-buffer and get it to compare correctly to gl_FragCoord.z.
I want to do:
vec2 otherZ=texture(otherZ,uv).r;
if (otherZ>=gl_FragCoord.z) discard;
But, I am not getting consistent results. Do I have to run some kind of conversion on the value in the other z-buffer? Is gl_FragCoord.z different from the value that ends up in the z-buffer?
Edited to add: This is the format of my stand-alone depth buffer:
glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH24_STENCIL8,mWidth,mHeight,0,GL_DEPTH_STENCIL,GL_UNSIGNED_INT_24_8, NULL);