OpenGL How to write to stencil buffer when stencil test fails and depth test succeeds?

Question

I'm trying to implement an effect in my program where i render the silouette of an object when it's behind other objects, similar to how it's done here: https://i.pinimg.com/originals/87/8f/c4/878fc47a5bd6d62dfbaec3520cb4d9f5.jpg

What i want is to write to the stencil buffer in the fragments of a cube object that pass the stencil test but don't pass the depth test.

This is my implementation:

glBindFramebuffer(GL_FRAMEBUFFER, this->screenFBO);
glEnable(GL_DEPTH_TEST);
glStencilOp(GL_KEEP,    // stencil fail
            GL_REPLACE, // stencil pass, depth fail
            GL_KEEP);   // stencil pass, depth pass

glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glStencilMask(0x00);
glDisable(GL_STENCIL_TEST);
this->renderEntity(entityBuffer[0]); // skybox
glEnable(GL_STENCIL_TEST);

glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0xFF);

this->renderEntity(entityBuffer[3]); // white cube

glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00);
// render rest of the models

The framebuffer is complete, the stencil buffer works as intended, since i already tested with a tutorial from https://learnopengl.com/Advanced-OpenGL/Stencil-testing

But the way i have it setup, it doesn't render anything to the stencil buffer, which i can check through RenderDoc.

Code Output (the cube stays behind the sphere)

What am i doing wrong? and thanks in advance for the time!

Answer 1

I finally got it fully working! The solution for me was to first draw everything in the color buffer and depth buffer, but not on the stencil buffer (basically just enabling depth test before drawing and setting the glStencilMask(0) ) Then draw the item i want to highlight making sure i write it in the stencil buffer by using glStencilMask(255) and making sure it passes the stencil test with glStencilFunc(GL_ALWAYS, 1, 255)

Then disable the depth test to render the shiny outline / shape to be rendered through other objects by using glStencilMask(0) and making sure that it wouldn't pass the stencil test when rendered on top of the other cube, by using glStencilFunc(GL_NOTEQUAL, 1, 255)

Render the shiny cube, re-enable the depth test and it works!

Also make sure that glStencilOp is set to: glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

This way the stencil buffer will be written only when the item writing to the stencil buffer isn't occluded by other objects (fails the depth test).

For me the biggest issue was figuring out why the stencil buffer would be updated even though my object would be occluded by other objects, and what i didn't think of was that i was rendering it before other objects, so there was nothing to occlude it when it was rendered! so make sure you render your scene first, and then your objects that is gonna have this property, so you can utilize the glStencilOp functionalities

glBindFramebuffer(GL_FRAMEBUFFER, this->screenFBO);
glEnable(GL_DEPTH_TEST);
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP,    // stencil fail
            GL_KEEP, // stencil pass, depth fail
            GL_REPLACE);   // stencil pass, depth pass

glStencilMask(255);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glStencilMask(0);

this->renderEntity(entityBuffer[0]); // skybox

glStencilMask(0);
glStencilFunc(GL_ALWAYS, 1, 255);
this->renderEntity(entityBuffer[6]); // sphere
this->renderEntity(entityBuffer[1]);
this->renderEntity(entityBuffer[2]);
this->renderEntity(entityBuffer[4]);
this->renderEntity(entityBuffer[5]);
this->renderEntity(entityBuffer[7]);
this->renderEntity(entityBuffer[8]);
this->renderEntity(entityBuffer[9]);
this->renderEntity(entityBuffer[10]);
this->renderEntity(entityBuffer[11]);
this->renderEntity(entityBuffer[12]);

glStencilFunc(GL_ALWAYS, 1, 255);
glStencilMask(255);
this->renderEntity(entityBuffer[3]); // normal cube

glStencilMask(0);
glStencilFunc(GL_NOTEQUAL, 1, 255);

glDisable(GL_DEPTH_TEST);
entityBuffer[3]->setShader(5);
entityBuffer[3]->setScale(glm::vec3(1.05));
this->renderEntity(entityBuffer[3]); // white cube
entityBuffer[3]->setScale(glm::vec3(1.0));
entityBuffer[3]->setShader(0);
glEnable(GL_DEPTH_TEST);

result can be seen here