webgl - camera position affecting spotlight

Question

I have been trying to implement a spotlight in my project. It works but at some point I noticed that when moving the camera around, the light was following the camera, and this is not what I wanted.

I found a way to fix the spotlight position so that it doesn't follow the camera anymore (multiplying it by the viewMatrix), but not the spotlight direction.

If I multiply the spotlight direction by the viewMatrix as in the following code, the spotlight appears to be moving in an unexpected way when I move the camera.

In my javascript file I compute the following:

viewMatrix = lookAt(viewerPos, at, up);

vertex shader:

    vec3 pos = (uModelViewMatrix*aPosition).xyz;
    N = normalize(uNormalMatrix*aNormal.xyz);
    E = -normalize(pos);    


    vec3 cameraToLight = (uViewMatrix*uSpotLightPosition).xyz;
    L = normalize(cameraToLight - pos );

    vec3 cameraToDirection = (uViewMatrix*uSpotLightDirection).xyz;
    lightDir = normalize(cameraToDirection - cameraToLight);
    gl_Position = uProjectionMatrix * uModelViewMatrix * aPosition; 

fragment shader:

        vec3 H = normalize( L + E );
        float Kd = max(dot(L, N), 0.0);
        float Ks = pow(max(dot(N, H),0.0),uShininess);
    
        vec4 ambient = uAmbientProduct;
        vec4 diffuse = Kd*uDiffuseProduct;
        vec4 specular = Ks*uSpecularProduct;            

        if( dot(L, N) < 0.0 )
            specular = vec4(0.0, 0.0, 0.0, 1.0);
        
        if(dot(normalize(lightDir), -normalize(L))>=uSpotlightOpeningAngle)
            fColor =  ambient+ diffuse;
        else 
            fColor =  vColor;