OpenGL texture pixel mapping stripe glitches (not border)

Question

I'm working on OpenGL Bayer demosaicing algorithms and although I've managed to make the techniques seemingly work, I'm getting occasional glitches in the output and I can't seem to figure out why. See images here: https://i.stack.imgur.com/5ik3v.jpg. The stripes occur only on certain scales of the surface, on most scales (maybe 99%) there are no glitches.

Vertex shader:

vec2 invTexSize = 1.0/texSize;
centerCoord = floor(vertexIn*texSize) + 0.5;
centerTexCoord = centerCoord * invTexSize;

Xoffsets = centerTexCoord.x + vec2(-invTexSize.x, invTexSize.x);
Yoffsets = centerTexCoord.y + vec2(-invTexSize.y, invTexSize.y);

gl_Position = mvp_matrix * vec4(vertexIn.xy,0.0,1.0);

Fragment shader:

float center = texture2D(source, centerTexCoord).r;

vec4 diags = vec4(
    texture2D(source, vec2(Xoffsets.x, Yoffsets.x)).r,          // (-1,-1)
    texture2D(source, vec2(Xoffsets.x, Yoffsets.y)).r,          // (-1, 1)
    texture2D(source, vec2(Xoffsets.y, Yoffsets.x)).r,          // ( 1,-1)
    texture2D(source, vec2(Xoffsets.y, Yoffsets.y)).r);         // ( 1, 1)

vec2 horiz = vec2(
    texture2D(source, vec2(Xoffsets.x, centerTexCoord.y)).r,    // (-1, 0)
    texture2D(source, vec2(Xoffsets.y, centerTexCoord.y)).r);   // ( 1, 0)

vec2 vertic = vec2(
    texture2D(source, vec2(centerTexCoord.x, Yoffsets.x)).r,    // ( 0,-1)
    texture2D(source, vec2(centerTexCoord.x, Yoffsets.y)).r);   // ( 0, 1)

float GREEN4 = (horiz.x + horiz.y + vertic.x + vertic.y)/4.0;
float DIAG4 = (diags.x + diags.y + diags.z + diags.w)/4.0;
float H2 = (horiz.x + horiz.y)/2.0;
float V2 = (vertic.x + vertic.y)/2.0;

// Branching
vec2 branch = mod(floor(centerCoord), 2.0);
gl_FragColor.rgb =  (   branch.y == 0.0) ?
                    ((      branch.x == 0.0) ?
                                vec3(center,GREEN4,DIAG4) :
                                vec3(H2,center,V2)) :
                    ((      branch.x == 0.0) ?
                                vec3(V2,center,H2) :
                                vec3(DIAG4,GREEN4,center));

Vertices are defines as

float vertices[] = {0.0f,   0.0f,
                    0.0f,   1.0f,
                    1.0f,   0.0f,
                    1.0f,   1.0f};

Filters are set to GL_NEAREST. texSize is the size of the uploaded texture image in pixels. Just for convenience(?) I use the same vertex and texture coordinates [0,1] and the mvp is configured ortho accordingly to maintain the aspect ratio. I'm using Qt5.5 MSVC32, and the glitches have been reproduced on very different hardware.

I've played around with many different mapping techniques (adjusting the texture coordinates etc.). Some of them seem to produce better results (less often glitches) than others and because of that I think the reason might be something like not accurate enough texture coordinates. At one point at least I thought the problem was gone when I didn't use mvp to maintain the aspect ratio, but was I just imagining, could that make a difference?

Any idea what might be causing this?

Some code owes credit to graphics.cs.williams.edu/papers/BayerJGT09/

The sample CFA image is from www.cs.unc.edu/~lazebnik/spring09/assignment1.html

UPDATE 6.12.2015: Keeping my fingers crossed here but I think I made good progress by fetching using unnormalized texture coordinates

texelFetch(source, ivec2(x, y), 0).r

instead of texture/texture2D. I reverted all funky texture coordinate adjustments and fed [0, size] vertices in:

centerCoord = vertexIn;

Initial tests show good results—we'll see if it's premature. And even if this was the solution, I still don't understand why I wasn't able to hit the texels with normalized coordinates.