Messages in azure queue vanish

Question

I hope someone here can shed a little light on an issue that I have encountered twice now. I have an ERP system in which invoices are created, when these invoices are ready to be sent, I then transfer them to our invoicing system through a cron scheduled job. When they are sent to the end customer from the invoicing system, it fires a webhook to an azure http trigger function which puts a message (the invoiceId) in a queue. I then have a queue trigger that picks these up and updates our ERP system so that the invoice can no longer be altered. This works great 90% of the time.

Last week we sent 12 invoices over to the invoicing system and our bookkeeper sent these to the customers. When checking this morning 2 of them were not updated with a "sent" status in our ERP system. So I check the queue trigger and can see that there is no invocation for the two invoices in question (under the function -> monitor). So I check the poison queue and it's not present there either nor is it in the real queue. Finally I check the Http trigger invocations and I can see that there is an invocation for the two invoices in question and I have a log around where it enqueues the message which have been properly logged with no errors.

So what strikes me as odd, is that for the other 10 invoices this went fine and nothing was missing. But for these two the queue message just seem to vanish. Does anyone have any ideas?

Sharing my two functions for adding to the queue and updating our ERP system.

HttpTrigger

[FunctionName(nameof(InvoiceBooked))]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)]
            HttpRequest req, ILogger log)
        {
            try
            {
                log.LogInformation("Invoice Booked from VS. C# HTTP trigger function processed a request.");
                string invoiceBookedId = req.Query["invoiceId"];
                log.LogInformation($"Invoice Booked. RequestBody: {invoiceBookedId}");

                if (string.IsNullOrEmpty(invoiceBookedId))
                {
                    log.LogError("Invoice Booked. Query was empty");
                    return new BadRequestResult();
                }

                // Get the connection string from app settings
                var storageAccountName = System.Environment.GetEnvironmentVariable("StorageAccountName", EnvironmentVariableTarget.Process);
                var storageAccountKey = System.Environment.GetEnvironmentVariable("StorageAccountKey", EnvironmentVariableTarget.Process);
                string connectionString =
                    $"DefaultEndpointsProtocol=https;AccountName={storageAccountName};AccountKey={storageAccountKey};EndpointSuffix=core.windows.net";

                // Instantiate a QueueClient which will be used to create and manipulate the queue
                var queueClient = new QueueClient(connectionString, AzureConstants.InvoiceBookedQueueName);

                // Create the queue
                await queueClient.CreateIfNotExistsAsync();
                log.LogInformation($"Invoice Booked. Enqueuing message: {invoiceBookedId}");
                if (await queueClient.ExistsAsync())
                {
                    var messageBase64 = System.Convert.ToBase64String(
                        System.Text.Encoding.UTF8.GetBytes(invoiceBookedId));

                    // Send a message to the queue
                    await queueClient.SendMessageAsync(messageBase64);

                    log.LogInformation($"Invoice Booked. Message enqueued: {invoiceBookedId}");
                }
            }
            catch (Exception e)
            {
                log.LogError(e, "Invoice Booked. Error when enqueueing booked invoice");
            }

            return (ActionResult)new OkResult();
        }

Queue Trigger

[FunctionName(nameof(InvoiceBookedQueueTrigger))]
        public static void Run([QueueTrigger(AzureConstants.InvoiceBookedQueueName, Connection = "QueueStorage")]string queueItem, ILogger log)
        {
            log.LogInformation($"InvoiceBookedQueueTrigger. C# Queue trigger function processed: {queueItem}");

            var erpService = new ERPService(log, System.Environment.GetEnvironmentVariable("APIKey", EnvironmentVariableTarget.Process));

            int.TryParse(queueItem, out var invoiceId);

            log.LogInformation($"invoiceId is: {invoiceId}");

            var success = erpService.SetInvoiceBooked(invoiceId);

            if(!success)
                throw new WebException("There was a problem updating the invoice in erp");
            
        }

I seem to have found some additional information. For some reason, the job host sometimes stops. I noticed today, where I was manually entering some ids that had vanished, that some went through, but one again vanished. In the trace log I could see that right around the time where it was supposed to fetch the queue item, the job host was stopped. What strikes me as odd is that the message is dequeued, no logs ar written about this at all. If I put the message in the queue when it is started, everything works fine. Any one got any ideas? I have added the log for you to look through

The log file can be downloaded here: https://1drv.ms/t/s!AotNYJ6EYJBWiRysY93fP2ODdFVX

Answer 1

Together with Microsoft Support I found that when running on a Consumption Plan there were issues that it may somehow lose the queue item. MS support was unable to explain the issue, but changing to an App Service Plan has completely remedied the issue.