SignalR + Redis in cluster not working

Question

Background

When running the website with single application instance (container) - SignalR is working perfectly.

When scaling out to more instances (>1), it throws errors and is just not working. I looked for an explanation on the internet and found that I need to configure my Signalr to work in cluster. I choose Redis as my backplane.

I worked through many tutorials on how to do it right and it's just not working for me.

Environment

I working asp.net core v2.1 hosted in Google cloud. The application is deployed as Docker container, using Kestrel + nginx. The docker container runs in a Kubernetes cluster behind a load balancer.

My Configuration

Startup.cs:

public class Startup
{
    public Startup(IConfiguration configuration,
        IHostingEnvironment hostingEnvironment)
    {
        Configuration = configuration;
        HostingEnvironment = hostingEnvironment;
    }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        // ...
        services.AddSignalR().AddRedis(options =>
        {
            options.Configuration = new ConfigurationOptions
            {
                EndPoints =
                    {
                        { ConfigurationManager.Configuration["settings:signalR:redis:host"], int.Parse(ConfigurationManager.Configuration["settings:signalR:redis:port"])}
                    },
                KeepAlive = 180,
                DefaultVersion = new Version(2, 8, 8),
                Password = ConfigurationManager.Configuration["settings:signalR:redis:password"]
            };
        });
        // ...
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env, IServiceProvider serviceProvider)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }
        else
        {
            app.UseExceptionHandler("/errors/general");
            app.UseHsts();
        }

        // nginx forward
        app.UseForwardedHeaders(new ForwardedHeadersOptions
        {
            ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
        });

        app.UseSignalR(routes =>
        {
            routes.MapHub<StatisticsHub>("/hubs/myhub");
        });
    }
}

In order to verify that the connection to Redis server succeeded, I checking the output window of the Kastrel:

Same behaviour (connected) found also on servers (2 replicas, not on development enviroment).

In order to verify that the Signalr is "really" using the Redis (not just connecting), I used redis-cli to connect the Redis server and found that:

From this I can understand that there are some "talks" on Redis.

I removed the website LoadBalancer (GCP) and deployed it agian. Now with Sticky-Session: ClientIP. This loadbalancer is routing requests to the different containers.

The only place that I didnt changed is the nginx configuration. I'm wrong?

The result

SignalR not working on the browser. These errors are from the browser console:

scripts.min.js:1 WebSocket connection to 'wss://site.com/hubs/myhub?id=VNxKLazEKr9FKM4GPZRDhA' failed: Error during WebSocket handshake: Unexpected response code: 404
scripts.min.js:1 Error: Failed to start the transport 'WebSockets': undefined
Error: Connection disconnected with error 'Error: Server returned handshake error: Handshake was canceled.'.
scripts.min.js:1 Error: Connection disconnected with error 'Error: Server returned handshake error: Handshake was canceled.'.
files.min.js:1 Uncaught (in promise) Error: Error connecting to signalR. Error= Error
...

Question

What I missed? What to check?