Azure app service plan running out of temp storage

Question

We are having a very peculiar problem. Running about 30 app services in P3v2 / P3v3 app service plan (with temp storage of 61/140gb respectively).

The temp storage is gradually filling up unless we change the plan (lower or up, which effectively restarts all services) or we restart all app services. Which is the recommended short-term remedy.

In the diagnostic blade Temp File Usage - the graph shows very limited data - only the "machine" which is taking up the space - in our case it's a single machine so we can only monitor spikes there. No information on which of the 30 services is using the temp storage, whatsoever.

We also tried some Kudu magic - iterating over all apps and executing commands so they can report the disk usage. All of them report very minimal usage (0.05-0.4gb used).

So we end up in a scenario where 130gb temp is used according to diagnostic logs. We run the scripts and apps report about 3-4gb temp used in total. Why the mismatch?

Some of the scripts we use (we fire them to the Kudu command api:

$apiUrl = "https://$($app.Name).scm.azurewebsites.net/api/command"
$command = "powershell -command ""[math]::Round((Get-ChildItem -Path C:\local -Recurse | Measure-Object -Property Length -Sum).Sum / 1GB, 2)"""

"du -sh D:\local\Temp"

One of my theories is that there are some locked files (i.e. created then deleted, but still locked by processes). But how can we be sure if no command is reporting data properly.

Another speculation is that we get some data hidden away from temp due to the SCM Separation config/flag - WEBSITE_DISABLE_SCM_SEPARATION Next step might be disabling separation and then running the commands, but it's a legacy mode and does not seem appropriate to enable in a prod environment.

I checked several references for ideas, but so far none lead me to a definitive conclusion: App service temp file limit How the azure app service file system works

Answer 1

You cannot increase the temp folder size in your Azure Web app as it has its limitation, the only option is to scale your App and various App service tiers has various Disk sizes available for the Web App. Refer this SO answer by Venkatesh Dodda

It seems like there might be temporary files and folders generated during normal job execution within your app services. Typically, these files are transient and get deleted before they're visible to users, but in rare cases, they might persist temporarily. These files aren't considered part of user storage or billed for, but they could cause issues if your job logic involves enumerating all files in a folder, leading to unexpected failures or errors due to these temporary files being present.

The WEBSITE_DISABLE_SCM_SEPARATION configuration can impact how certain files are stored or accessed within the app service. Disabling it temporarily for diagnostic purposes might be a valid approach, though as you've mentioned, it's not an ideal solution for a production environment.

If you want to clear your temp data without restarting your web app, You can refer this MS Q & A forum answer by Niharika Koneru

For Windows Web App:-

del /q/f/s %TEMP%*

For Linux Web App:-

rm -rf /tmp/*

After running these commands, Check if there are any temporary files that persists.

You can also get an insight into your Disk storage and resolve it by visiting Diagnose and solve problems of your Web app and check Disk usage and running processes:-

Reference:-

Understanding the Azure App Service file system · projectkudu/kudu Wiki (github.com)