OpenGL shader - overlaping multiple textures

Question

I am not able to find a pattern to draw textures one over other. I need to make result fragment color like:

tex1 + (1-tex1alpha)*tex2 + (1-tex1alpha-tex2alpha)*tex3

Not to mix textures , but to place one over other like layers in image editor.

Answer 1

I don't really use OpenGL, so the GLSL code might not be completely valid. But the shader you are looking for should look something like this. Feel free to correct any mistakes.

vec4 col;
col = texture2D(tex3, uv)
if(col.a == 1) { gl_FragColor = col; return; }
col = texture2D(tex2, uv)
if(col.a == 1) { gl_FragColor = col; return; }
col = texture2D(tex1, uv)
if(col.a == 1) { gl_FragColor = col; return; }

That is if the number of textures is fixed. If you have a variable number of textures, you could put them into a texture array and do something like:

vec4 col;
for(int i = nTextures - 1; i >= 0; --i)
{
    col = texture2DArray(textures, vec3(uv, i));
    if(col.a == 1)
    {
        gl_FragColor = col;
        return;
    }
}

Answer 2

You can write the formula you posted directly in to the shader. The following will work:

uniform sampler2D sampler1;
uniform sampler2D sampler2;
uniform sampler2D sampler3;

varying vec2 texCoords;

void main()
{
    vec4 tex1 = texture(sampler1, texCoords);
    vec4 tex2 = texture(sampler2, texCoords);
    vec4 tex3 = texture(sampler3, texCoords);
    gl_FragColor = tex1 + (1 - tex1.a) * tex2 + (1 - tex1.a - tex2.a) * tex3;
};

However the 3rd argument could turn negative, which would have undesired results. Better would be:

gl_FragColor = tex1 + (1 - tex1.a) * tex2 + max(1 - tex1.a - tex2.a, 0) * tex3;