glStencilFunc not working for some reason :(

Question

No matter what I tried, particles render outside the globe without touching the inside of the globe, I want it the other way around! no matter what parameters I changed (tried for 2 hours), its either particles render outside fine but inside not, or particles both outside and inside. I am getting a huge headache from this, can someone please help me solve this bug. I am using 1 bit for the stencilbuffer. GDEbugger shows the stencil buffer working fine.

glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_NEVER, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
globe->Use();
globe->SetWVP(&WVP, true);
Render(globe.obj");
glDisable(GL_STENCIL_TEST);

glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

RenderParticlesUsingTransformFeedback();

Answer 1

You are disabling the stencil test right before you render your particles.

Answer 2

glDisable(GL_STENCIL_TEST);

If you want to have RenderParticlesUsingTransformFeedback actually tested against the stencil buffer, you have to actually use that. Also, I assume that the "UsingTransformFeedback" part means that you generated these particle previously with TF. Because stenciling doesn't cull triangles, so it will not affect the generation of data with transform feedback.

In any case, your stencil code doesn't work because it can't work. You cannot use just a stencil buffer to ensure that particles are not rendered outside of a three dimensional volume. You can do it outside of a 2D volume. And you can partially do it for particles outside of a 3D volume, if you don't mind objects behind the sphere from being rendered.

But if you want to limit it to a 3D volume, you're going to have to use some shader logic, not the stencil buffer. You have to keep the closest point of the 3D volume, but you also need the farthest point. So you need to do some interesting depth buffer shenanigans.

When you render the particles, you need to take both of the depth buffers you used to capture the near and far values. Then, in your shader, you discard the fragment if the current depth is outside of that range.