Different GL shader results from different platform/hw(YUV to RGB)

Question

I wrote a WebGL shader that renders yuv420p picture supplied via 3 separate texture units(so it can also handle planar 422, 444 ...). You can run it from your browser here. And here's the original image I converted the yuv file from.

I created the YUV file with:

ffmpeg -i color-test.jpg -pix_fmt yuvj420p -f rawvideo color-test.yuv

I explicitly used yuvj420p instead of yuv420p to get full range values(0-255). So if I were to use this code on production, I'd add some uniforms to adjust to the colour range.

The Vertex Shader

attribute vec3 a_pos;
attribute vec2 a_uv;

uniform mat4 u_mvp;

varying vec2 v_uv;


void main () {
  gl_Position = u_mvp * vec4(a_pos, 1.0);
  v_uv = a_uv;
}

The Fragment Shader

precision mediump float;

uniform sampler2D u_texY;
uniform sampler2D u_texU;
uniform sampler2D u_texV;
uniform vec4 u_diffuse;

varying vec2 v_uv;

// YUV to RGB coefficient matrix.
const mat3 coeff = mat3(
  1.0, 1.0, 1.0,
  0.0, -0.21482, 2.12798,
  1.28033, -0.38059, 0.0
);


void main () {
  vec3 yuv = vec3(texture2D(u_texY, v_uv).a, texture2D(u_texU, v_uv).a, texture2D(u_texV, v_uv).a) * 2.0 - 1.0;
  vec3 colour = (coeff * yuv + 1.0) / 2.0;

  gl_FragColor = u_diffuse * vec4(colour, 1.0);
}

The coefficient matrix is that of BT.709 rec(this wiki page). u_diffuse is typically vec4(1.0, 1.0, 1.0, 1.0) unless you want to diffuse the picture with a certain colour.

Texture Upload

The textures are uploaded like this(line 233):

  gl.activeTexture(gl.TEXTURE0);
  gl.bindTexture(gl.TEXTURE_2D, tex.y);
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.ALPHA, rp.yuv.meta.width, rp.yuv.meta.height, 0, gl.ALPHA, gl.UNSIGNED_BYTE, picture.y);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
  gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);

CLAMP_TO_EDGE is used to work with any dimension(any dimension that's divisible by 2, that is) of image. Note that the textures are updated on every draw callback. This is intentional because normally we'd have to update the textures every time a picture is handed over from the video decoder.

The Problem

The most obvious problem would be that the rendered image is visibly different from the rendition of image viewer apps(i.e. web browser). Those programs probably software scale the images from YUV to RGB with CPU. And that's the most accurate way to convert, i guess. What's causing this exactly in my program? I've done my research, but I couldn't come to a clear explanation. Floating point precision?

The problem expands even further. The colours are different on different devices. For example, the woman looks slightly different on my android phone from on my PC. The same thing happens on Linux and Windows browsers. When I measured the colour of the black box in the bottom right corner, I got the "different black" across the platform/hw.

• Chrome on Linux: #060106
• Firefox on Linux: #010001
• Chrome on Android: #060006
• Chrome/Firefox/Oprea on Windows: #060106

Any way to mitigate this? And why can't I get the "true" black(#000000)?

Extra bit

I'm more than concern about this because some people do spot this difference. One day, as I was working in my office, this one client insisted that my video player looks different in colour than another company's program. No one knew why(didn't have the other program's source code). But it seemed that the other program rendered on RGB surface while my program rendered on a YUV DirectX surface. So they got to do the conversion from the CPU if that's the case. There could be many reasons as to why it looked different. Might have been colour range, bad coefficient that some Jimmy used... the list could go on.

Answer 1

You're making assumptions about where your chroma sample site is located; it's not always the middle of the UV texel.

You're linearly interpolating YUV samples which are still in non-linear gamma.

The color gamut of YUV is larger than the color gamut of RGB (you'll get values outside the 0 to 1 RGB range), so you need to either clip or remap the color values.

Not all graphics stacks handle RGB output gamma correctly, so you may find a mismatch between the gamma the GPU is emitting and how the display interprets it.

Also, are you sure that you are getting a BT.709 input? Chromacity constants differ for other formats, even if the pixel packing format in memory is the same (both in terms of recovering RGB, and the use of narrow or wide gamuts)