FallbackAsync Method stops REST calls

Question

public static IAsyncPolicy<HttpResponseMessage> GetSaveExceptionPolicy(
            IServiceProvider serviceProvider)
{

    return Policy
        .HandleResult<HttpResponseMessage>(x => x.StatusCode == HttpStatusCode.Forbidden)
        .FallbackAsync(new HttpResponseMessage(HttpStatusCode.OK),
            d =>
            {
                var mediator = serviceProvider.GetService<IMediatorService>();
                mediator.SaveFailedOnInvalidLock();
                return Task.CompletedTask;
            });
}

 serviceCollection
    .AddHttpClient<IMyClient, MyClient>(p =>
    {
        p.BaseAddress = baseAddress;
        p.Timeout = TimeSpan.FromMinutes(10);
        
    })
    .ConfigurePrimaryHttpMessageHandler(sp => new HttpClientHandler() { MaxConnectionsPerServer = 10})
    .AddPolicyHandler((serviceProvider, request) => GetSaveExceptionPolicy(serviceProvider));

this is the mediator call

public void SaveFailedOnInvalidLock()
{
    _logger.Error($"Data Point Saving failed");
    SaveFailedOnInvalidLock?.Invoke();
}

This MyClient Post methods are called very frequently. For example at a given second, there are 7 calls going on. and for each second there are 7 of them continues.

The application works fine until the Post method returns a "HttpStatusCode.Forbidden". When it returns a Forbidden, some of the next calls do not reach the REST end point.

Is this due to some connections not being available? eg: few connections are lost due to "fobidden" status and polly fallback policy? (because max limit is 10) ?

or is this due to Async and its timing such that the connections are still in use when the 2nd lot of calls come in?

thanks in Advance

Answer 1

I was able to solve this by returning the same http response message in the fallback action ( rather than creating a new HTTPResponseMessage)

I guess since i have MaxConnectionsPerServer = 10 and I'm using 7 of them at a given moment, if I return new HttpResponseMessage(HttpStatusCode.OK) there are some handlers that are not disposed or stuck.

This is my new policy

public static AsyncFallbackPolicy<HttpResponseMessage> GetDataPointSaveExceptionPolicy(
    IServiceProvider serviceProvider)
{
   return Policy 
       .HandleResult<HttpResponseMessage>(x=>
           x.StatusCode ==  HttpStatusCode.Forbidden)
       .FallbackAsync(
           (res, cx, ct) =>
           {
               var dataServiceMediator = serviceProvider.GetService<IDataServiceMediatorService>();
               dataServiceMediator.DataPointSaveFailedOnInvalidLock();

               return Task.FromResult(res.Result); 
           },
           (ct, cx) => Task.CompletedTask);
}

This returns Task.FromResult(res.Result) and there are no dispose issues and the calls to the MyClient continues to Reach REST end point.