My Azure service bus receiver closes due to Idle Timeout and I haven't found a way to fix it

Question

I have a pretty basic .NET 6 app which only goal is to receive messages from Azure Service Bus and handle them. When my app works, I am pretty happy with the result and it is answering the business logic I expect.

The thing is, after listening to a message and handling it, my Azure service bus receiver goes iddle and the link is closed.

Here are some logs from my deployed app (it is a docker image deployed on AKS through jenkins/argocd for information) :

info: Inventory_Create_Worker.API.Consumer.CreateInventoryConsumerService[0]

      Message received and handled: {"Title":"inventory creation event","Description":"Extract from excel file","Event":":CONFIDENTIAL_INVENTORY_DATA"}

info:  Azure.Messaging.ServiceBus[38]

      Receive Link Closed. Identifier: queue-inventory-creation-dev-xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx, 
      SessionId: , linkException: Azure.Messaging.ServiceBus.ServiceBusException: The link 'amqps://personaldevservicebus-development.servicebus.windows.net/-f1a43db3;0:5:6:source(address:/queue-inventory-creation-dev,filter:[])'
      is force detached. Code: aggregate-link1318524517. Details: AmqpMessagePartitioningEntityMessageCache.IdleTimerExpired: Idle timeout in Seconds: 900. (GeneralError).
      For troubleshooting information, see https://aka.ms/azsdk/net/servicebus/exceptions/troubleshoot..

Here is how I inject into my ioc IAzureClientFactory :

            services.AddAzureClients(clientsBuilder =>
            {
                clientsBuilder.AddServiceBusClient(connectionString:azureQueueConfig.ConnectionString)
                    .WithName(ConfirmationCreateServiceBusClientName)
                    .ConfigureOptions(options =>
                    {
                        options.RetryOptions.Delay = TimeSpan.FromSeconds(2);
                        options.RetryOptions.MaxDelay = TimeSpan.FromMinutes(2);
                        options.ConnectionIdleTimeout = TimeSpan.FromMinutes(5);
                        options.RetryOptions.MaxRetries = 5;
                    });
            });

At first I didn't have any .ConfigureOptions and I feel like my issue is probably inside it.

Here is my background service which consumes messages from Azure Service bus :

using Shared;
using Shared.AzureServiceBus;
using Shared.Interfaces;
using Shared.Interfaces.Infrastructure;
using Shared.Interfaces.Provider;
using Shared.Interfaces.Service;
using Shared.Models.Config;
using Shared.RabbitMq;
using Shared.Services;
using Inventory_Create_Worker.Application.Common.Interfaces.Authentication;
using Inventory_Create_Worker.Infrastructure.Authentication;
using Inventory_Create_Worker.Shared.Constants;
using Microsoft.Extensions.Azure;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using static Inventory_Create_Worker.Shared.Constants.AzureConstant;


    public class CreateInventoryConsumerService : BackgroundService
    {
        private readonly ILogger<CreateInventoryConsumerService> _logger;
        private ServiceBusReceiver _serviceBusReceiver;
        private ServiceBusClient _serviceBusClient;
        private readonly IAzureClientFactory<ServiceBusClient> _azureClientFactory;
        private readonly AzureQueueConfig _azureQueueConfig;
        private readonly IServiceProvider _serviceProvider;

        public CreateInventoryConsumerService(IAzureClientFactory<ServiceBusClient> serviceBusClientFactory,
            ILogger<CreateInventoryConsumerService> logger,
            IOptions<AzureQueueConfig> azureQueueOptions, IServiceProvider serviceProvider)
        {
            _logger = logger;
            _serviceProvider = serviceProvider;
            _azureClientFactory = serviceBusClientFactory;
            _azureQueueConfig = azureQueueOptions.Value;
            _serviceBusClient = _azureClientFactory.CreateClient(name: AzureConstant.InventoryCreateServiceBusClientName);
            _serviceBusReceiver = _serviceBusClient.CreateReceiver(queueName: _azureQueueConfig.QueueName);
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                var receivedMessage = await _serviceBusReceiver.ReceiveMessageAsync(cancellationToken: stoppingToken);
                if (receivedMessage != null)
                {
                    using (var scope = _serviceProvider.CreateScope())
                    {
                        var serviceProvider = scope.ServiceProvider;
                        var mediator = serviceProvider.GetRequiredService<IMediator>();
                        Console.WriteLine("BEGINNING TO CONSUME SERVICE");

                        var createInventoryEvent = Encoding.UTF8.GetString(receivedMessage.Body);
                        Console.WriteLine("var createInventoryEvent = Encoding.UTF8.GetString DONE");

                        CreateInventoryFromEventCommand command = new(createInventoryEvent);
                        Console.WriteLine(
                            "CreateInventoryFromEventCommand command = new(createInventoryEvent); DONE");

                        await mediator.Send(command, stoppingToken);
                        Console.WriteLine("_mediator.Send(command) DONE");

                        _logger.LogInformation("Message received and handled: " + createInventoryEvent);
                        Console.WriteLine("Message received and handled: " + createInventoryEvent);
                    }
                }
            }
            catch (Exception ex)
            {
                _logger.LogError($"Message could not be executed in Inventory_Create_Worker.CreateInventoryConsumerService.ExecuteAsync, Exception raised: {ex.Message} )");
                Console.WriteLine($"Message could not be executed in Inventory_Create_Worker.CreateInventoryConsumerService.ExecuteAsync, Exception raised: {ex.Message}");
                RecreateServiceBusInstances();
            }
        }
        /// <summary>
        /// When our service bus crashes, we need to recreate it.
        /// If not we are not listening to Azure Service bus events anymore and our Inventory consumer service is not guaranteed anymore
        /// </summary>
        private void RecreateServiceBusInstances()
        {
            try
            {
                _logger.LogInformation("Recreating a new ServiceBusReceiver and a new  ServiceBusClient instance");
                Console.WriteLine("Recreating a new ServiceBusReceiver and a new ServiceBusClient instance");
                _serviceBusClient =
                    _azureClientFactory.CreateClient(name: AzureConstant.InventoryCreateServiceBusClientName);
                _serviceBusReceiver = _serviceBusClient.CreateReceiver(queueName: _azureQueueConfig.QueueName);
            }
            catch (Exception ex)
            {
                _logger.LogError($"Failed to recreate ServiceBus instances: {ex.Message}");
                Console.WriteLine($"Failed to recreate ServiceBus instances: {ex.Message}");
            }
        }
    }

Answer 1

Thanks @DavidG, @Jesse Squire for your inputs.

Your CreateInventoryConsumerService looks good, and it's designed to handle messages from Azure Service Bus using a background service in a .NET application.

var receivedMessage = await _serviceBusReceiver.ReceiveMessageAsync(cancellationToken: stoppingToken);

• Here we need to check that stoppingToken passed to ReceiveMessageAsync is effectively linked to the cancellation token of the BackgroundService.

Here I have updated configuration to set the ConnectionIdleTimeout to a longer duration.

services.AddAzureClients(clientsBuilder =>
{
    clientsBuilder.AddServiceBusClient(connectionString: azureQueueConfig.ConnectionString)
        .WithName(ConfirmationCreateServiceBusClientName)
        .ConfigureOptions(options =>
        {
            options.RetryOptions.Delay = TimeSpan.FromSeconds(2);
            options.RetryOptions.MaxDelay = TimeSpan.FromMinutes(2);
            options.ConnectionIdleTimeout = TimeSpan.FromMinutes(15); // Set a longer idle timeout
            options.RetryOptions.MaxRetries = 5;
        });
});

Received messages:

Azure ServiceBus: