HLSL Normal mapping - light reflections in the shadows

Question

I just recently got into HLSL and I'm currently following Ben Cloward's DVD tutorials on youtube. The tutorial itself is a bit dated but for the most part I can manage when some functions don't work.

However, I noticed that when I try to bring normal maps in my shader, it looks like the light is creating highlights with the normal where the light values should all be 0. I'm not sure if it comes from the code itself or if it's due to the older drivers I'm using so I'm a bit at a loss for answers right now.

Specs

Key	Value
Application	3DS Max 2023.2.2
Viewport Driver	Legacy Direct3D (Direct3D 9.0 (NVIDIA GeForce RTX 3080))
Vertex Shader Compiler	vs_1_1
Pixel Shader Compiler	ps_2_0

Code snippets

You can find a full version of the file here, but here's a few snippets that might contain the problem:

Setting the texture

texture normalMap : NormalMap
<
    string name = "default_bump_normal.dds";
    string UIName = "Normal Texture";
    string TextureType = "2D";
>;

[...]

sampler2D normalMapSampler = sampler_state
{
    Texture = <normalMap>;
    AddressU = Wrap;
    AddressV = Wrap;
    MinFilter = linear;
    MagFilter = linear;
    MipFilter = anisotropic;
};

Structs

// input from application
    struct a2v {
    float4 position     : POSITION;
    float2 texCoord     : TEXCOORD0;
    float3 normal       : NORMAL;
    float3 binormal     : BINORMAL;
    float3 tangent      : TANGENT;
};


// output to fragment program
struct v2f {
    float4 position        : POSITION;
    float2 texCoord        : TEXCOORD0;
    float3 lightVec        : TEXCOORD1;
    float3 worldNormal     : TEXCOORD2;
    float3 worldBinormal   : TEXCOORD3;
    float3 worldTangent    : TEXCOORD4;
};

Vertex Shader

v2f v(a2v In, uniform float4 lightPosition)
{
    v2f Out;                                                            //create the output struct
    Out.worldNormal = mul(In.normal, WorldInverseTranspose).xyz;        //put the normal in world space pass it to the pixel shader
    Out.worldBinormal = mul(In.binormal, WorldInverseTranspose).xyz;    //put the binormal in world space pass it to the pixel shader
    Out.worldTangent = mul(In.tangent, WorldInverseTranspose).xyz;      //put the tangent in world space pass it to the pixel shader
    float4 worldSpacePos = mul(In.position, World);                     //put the vertex in world space
    Out.lightVec = lightPosition - worldSpacePos;                       //create the world space light vector and pass it to the pixel shader
    Out.texCoord.xy = In.texCoord;                                      //pass the UV coordinates to the pixel shader
    Out.position = mul(In.position, WorldViewProjection);               //put the vertex position in clip space and pass it to the pixel shader
    return Out;
}

Pixel Shader

float4 f(v2f In,uniform float4 lightColor) : COLOR
{
    //fetch the diffuse and normal maps
    float4 ColorTexture = tex2D(diffuseMapSampler, In.texCoord);
    float4 normal = tex2D(normalMapSampler, In.texCoord)*2-1;
    normal = normal * float4(.5,.5,1,1);

    float3 Nn = In.worldNormal;
    float3 Bn = In.worldBinormal;
    float3 Tn = In.worldTangent;

    float3 N = (normal.z * Nn) + (normal.x * Bn) + (normal.y * -Tn);    // In Y-up env, formula would be (normal.z *Nn) + (normal.x * Tn) + (normal.y * -Bn)

    N = normalize(N);

    //create lighting vectors - view vector and light vector
    float3 L = normalize(In.lightVec.xyz);                              //the light vector must be normalized here so all vectors will be normalized
  
    //lighting
  
    //ambient light
    float4 Ambient = AmbientColor * ColorTexture;                       //To create the ambient term, we multiply the ambient color and the diffuse texture
  
    
    //diffuse light
    float4 diffuselight = saturate(dot(N, L)) * lightColor;             //To get the diffuse light value we calculate the dot product between the light vector and the normal                               
    float4 Diffuse = DiffuseColor * ColorTexture * diffuselight;        //To get the final diffuse color we multiply the diffuse color by the diffuse texture

    //return Diffuse + Ambient;
    return saturate(dot(N,L)) * lightColor;
}

---

I tried fixing my vertex shader outputs, without much of a difference.

I also tried messing with some values, but the only thing that got me close to what I want is that zooming in makes the effect stronger, while zooming out makes it attenuated...

Other than that, I tried looking more info on the subject, but I can't seem to find anyone having my issue. I couldn't find something relevant in the microsoft documentation, although I didn't read every articles as it was giving me 20 000 results...

Anyone have an idea of what might be causing this (or where I might find my solution)?