I am developing a service which picks up messages from an Azure Service Bus Queue, does some minimal processing/validation on it, and then saves it to a file and some metadata to a database. This service is running on AKS (we are using the dotnet 6 base docker image) and it has a mounted volume which is connected to an Azure File share (Using Azure Disk CSI). The volume of messages we will see can be upwards of tens of thousands per minute so our service is continuously running and processing these messages. It has been working well, but I have noticed an issue in testing recently where every once in a while I will see a flurry of System.IO.IOException's and a corresponding decrease in the throughput of processing these messages. Specifically this is the message I see:
The process cannot access the file <filepath> because it is being used by another process.
I understand what this implies, the file is being used by some other process and is therefore locked. However, in my particular context each message will get its own file. The directories can be shared, so I guess that could be an issue, but I would not suspect that to actually cause the bottleneck I'm seeing and I think the code I wrote for creating the directories should handle this. I suppose it could be possible that some other process is listening to the queue and also got the message at the same time (I'm using ReceiveAndDelete ReceiveMode). However, upon reading and in my other testing I'm pretty sure that the SDK I'm using from Microsoft and/or the ServiceBus itself ensures that only one recipient gets the message. Addtionally, if multiple processes were getting the message I would be seeing multiple errors or the data would be saved eventually. However, I'm not seeing any data for the messages that have the exceptions logged in the files or the database. At this point I am having a hard time boiling this down to a specific question, but I'll give it a try. Is there some sort of throttling that could be happening on the File Share that could lead to this error on a mounted volume? Are there any other logs or traces I can look at for the File Share to suss out what is happening to the files that are associated with these errors?
For reference, here's the code that is being used to create the file (and it's directories)
public void Store(string relativeFilePath, string content)
{
var startDateTimeUtc = DateTimeOffset.Now.ToString();
var fullPath = Path.Combine(fileStorageProviderCredentials.Paths.Path, relativeFilePath);
try
{
var directory = Path.GetDirectoryName(fullPath);
if (!Directory.Exists(directory))
{
CreateDirectory(directory);
}
WriteFile(fullPath, content);
}
catch (Exception e)
{
var endDateTimeUtc = DateTimeOffset.Now.ToString();
if (e.Message.Contains("directory"))
{
throw new FileStorageProviderSaveException("Start Time: " + startDateTimeUtc + ", End Time " + endDateTimeUtc + ", error when creating directory" + fullPath, e);
}
else
{
throw new FileStorageProviderSaveException("Start Time: " + startDateTimeUtc + ", End Time " + endDateTimeUtc + ", error when writing file" + fullPath, e);
}
}
}
private void CreateDirectory(string directory)
{
var startDateTimeUtc = DateTimeOffset.Now.ToString();
var dir_created = false;
var retries = 0;
while (dir_created == false && retries < 10)
{
try
{
var dir = Directory.CreateDirectory(directory);
dir_created = true;
}
catch (Exception e)
{
Thread.Sleep(100 * retries); //Sleep for 100 milliseconds
if (retries == 19)
{
var endDateTimeUtc = DateTimeOffset.Now.ToString();
throw new FileStorageProviderSaveException("Start Time: " + startDateTimeUtc + ", End Time " + endDateTimeUtc + ", error creating directory file " + directory, e);
}
}
retries++;
}
}
private void WriteFile(string fullPath, string content)
{
var startDateTimeUtc = DateTimeOffset.Now.ToString();
var fileCreated = false;
var retries = 0;
while (!fileCreated && retries < 20)
{
try
{
File.WriteAllText(fullPath, content, Encoding.UTF8);
fileCreated = true;
}
catch (Exception e)
{
Thread.Sleep(100 * retries); //Sleep for 100 milliseconds
if (retries == 19)
{
var endDateTimeUtc = DateTimeOffset.Now.ToString();
throw new FileStorageProviderSaveException("Start Time: " + startDateTimeUtc + ", End Time " + endDateTimeUtc + ", error writing file " + fullPath, e);
}
}
retries++;
}
}
I'm not sure this is helpful, but here's a graph of the "performance" of the file share during this time:
This graph also shows the file creation time (I was tracking this and then logging it in code)
