Distort shadow using normal map

Question

I'm attempting to distort a "shadow" based on a normal map of the background, which I have access to in my shader.

Video

You can see here that I have a character hopping in front of a normal circle (while it looks off, this is just because of in-engine lighting; I have the (mostly) pure normal map under the hood which I again do have access to in this shader.)

However, for as much as I've used normal maps, I've never had to directly play around with them to do something like this, so I'm a bit at a loss. I've done a good amount of googling, but I'm afraid I don't even know the correct terminology for what I'm trying to do.

I'm working in an OGL-like shading language (Godot Shader Language to be specific), but even pseudocode would be good enough to set me on the right track. Much appreciated.

I've tried, misguidedly I feel, to use the dot product between the normal map and the light direction and just plugging that in where I thought it'd do something, though I never really expected it to work. I was more just throwing stuff at the wall. Originally, I was just using a height map, but I realized that that was only getting me distortion in one direction and resulting in very jank renders depending on the object.

Shader code (this is happening in Godot's Light processor)

vec4 heightTex = texture(bg_height, UV) + depthBrightness;
float depth = heightTex.x;
// (1.0 - depth): b/c LightDirection is in opposite direction, we need to 
// "flip" the depth.
// The mix call is to reinterpret the resulting depth to a smaller range
// [0,1] is too large and results in huge shadows even at depths of .7
vec4 normalTex = texture(bg_normal, UV);
vec2 sample = UV + (LIGHT_DIRECTION.xy * (mix(0.0, depthRemap, (1.0 - depth)) + depthMod));
vec4 grab = texture(TEXTURE, sample);
vec4 d4 = abs(grab - vec4(.5));
float d = max(max(d4.r, d4.g), d4.b);
if(d >= 0.1) {
    LIGHT = LIGHT_COLOR;
}
else {
    LIGHT = vec4(0.0);
}

There's additional code stripped out that handles precision as well as the fact that giving the sample the raw unadjusted r value is too much. But should more or less logically convey the process correctly. At the moment, I'm not doing anything with the normal map but you can assume I can grab it with something like

normalTex = texture(normalMap, UV)

And can modify the normals and UVs at will.

Also of note, the involved textures are composed from viewports. I render foreground and background elements to their own respective viewports to achieve this, and that's also how I get the composed height and normal maps.

Answer 1

If you have a height for the light, you can define a line from the light source to the current fragment, in 3D.

So, the position of the light in 3D is its 2D position augmented with the height of the light. I'm going to suggest to add an uniform offset you can add to the height of the light so you can tweak the effect.

Similarly, you augment the 2D position of the fragment with its height from the height map to get its 3D position.

And we need them in the same coordinante space. I'll take the assumption that we are augmenting on z.

Now you can get the direction and distance of the light in 3D:

vec3 light3D = fragmentPosition3D - lightPosition3D;
vec3 lightDir3D = normalize(light3D);

We want to find out where it intercept the foreground plane… I'll guess that the height of the foreground plane is an uniform for you to tweak, because I don't think the assumption that it is at height zero is a correct.

So, the points of the light ray can be defined in parametric form like this:

lightPosition3D + lightDirection3D * t

Which expands to this:

lightPosition3D.x + lightDir3D.x * t
lightPosition3D.y + lightDir3D.y * t
lightPosition3D.z + lightDir3D.z * t

And we are looking for a t such that:

lightPosition3D.z + lightDir3D.z * t = foregroundHeight

Wait, lightPosition3D.z is lightHeight:

lightHeight + lightDir3D.z * t = foregroundHeight

Solve for t:

t = (foregroundHeight - lightHeight) / lightDir3D.z

Replace it in here:

lightPosition3D + lightDirection3D * t

We get:

lightPosition3D + lightDir3D * (foregroundHeight - lightHeight) / lightDir3D.z

And that is where you need to sample the foreground. I don't know if you need an additional transformation, but in essence the xy of that vec3 should give you the UV to sample the texture.