StackExchange.Redis Timeout Exceptions (ASP.NET Core 6 MVC)

Question

We have an ASP.NET Core 6 MVC application that is deployed across multiple pods via K8s, and is using a single Azure Redis instance between those pods. On the .NET side, we are using StackExchange.Redis with the IDistributedCache interface.

Currently, we are seeing these types of Exceptions within our pods:

Type: StackExchange.Redis.RedisTimeoutException
      Source: StackExchange.Redis
      Message: Timeout performing HMGET (5000ms), next: HMGET <KEY NAME>, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: <SERVER>, mc: 1/1/0, mgr: 10 of 10 available, clientName: <CLIENT NAME>, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
      Stack Trace:    at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817
         at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54
         at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 321
         at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members)
         at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData)
         at Microsoft.Extensions.Caching.Distributed.DistributedCacheExtensions.GetString(IDistributedCache cache, String key)
         at Infrastructure.ExternalServices.CacheService.GetCacheValue[T](String key) in /src/Infrastructure/ExternalServices/CacheService.cs:line 49

Our Redis registration code looks like:

private static Lazy<ConnectionMultiplexer> _lazyConnection;

public static IServiceCollection AddInfrastructure(this IServiceCollection services,
    RedisConfiguration redisConfiguration)
{
    _lazyConnection = new Lazy<ConnectionMultiplexer>(() => {
        return ConnectionMultiplexer.Connect(redisConfiguration.ConnectionString);
    });
}

private static IServiceCollection AddRedisCache(this IServiceCollection services, RedisConfiguration redisConfiguration)
{
    services.AddStackExchangeRedisCache(options =>
    {
        options.Configuration = redisConfiguration.ConnectionString;
        options.InstanceName = redisConfiguration.InstanceName;
    });
    services.AddDataProtection().PersistKeysToStackExchangeRedis(_lazyConnection.Value, "DataProtection-Keys");
    return services;
}

Where AddRedisCache is called in Program.cs.

I am stumped on this one. To me it looks like threading shouldn't be an issue. Could this simply be due to a slow connection to our Redis instance? Should we be doing something differently in with ConnectionMultiplexer for a multi-instance environment? Does anything else stand out as problematic?

Any help is appreciated!

Tried increasing thread count to combat any threadblocking issues - did not work.

Tried using the Lazy connection - did not work.