BizTalk Server 2020 - Temp Files not getting cleaned within InProcess User Context

Question

We are using BizTalk Server 2020 with SFTP Adapter (WinSCP Version 5.21.5). Multiple Receive Locations and other Artefacts are running under this user.

Unfortunately our Admin contacted me that the Temp file directory of this user is increasing quickly (Location: C:\Users\<InProcess-User>\AppData\Local\Temp) – currently we are talking about a storage capacity of 30 GB. It seems like most of the files are related to the SFTP Adapter and used for tracking sessions during the WinSCP connections.

The InProcess user has Full control permissions to this folder.

I have following questions to this behavior

1. Why are the files not getting cleaned up?
2. Is there are setting which can be set up for this cleanup? If yes, where can I find it?
3. How can I avoid BizTalk following this behavior?
4. If there is no default functionality for it, what would be the best way to cleanup these files? Having a scheduled task, which cleans up files, older than x days?

My assumption is that directly after closing the connection, the SFTP adapter functionality should be responsible deleting that file, is that correct? Additionally I installed the CU5, hoping that it was a problem with the WinSCP version.

Answer 1

From my blog post Max worker threads for SFTP, MQSC or DB in BizTalk there are some recommendations to avoid issues with the SFTP adapter (amended with the fix for the above issue).

Recommendations for SFTP Receive Locations

1. Have a dedicated host for receive locations using SFTP
2. Increase the Maximum Worker threads setting to 500 on that host instance
3. Check how frequently you poll (the default is 5 seconds), increase the delay if that is not needed.
4. Put a schedule on to only poll during the periods you need.
5. Disable message body tracking if it is not needed.
6. For BizTalk 2016 ensure that FIX: Shared temporary files may never be deleted if BizTalk, WinSCP and SFTP server remain active installed (Cumulative Update 7 or up) or try reducing the Maximum Connection Reuse Time if there is an issue with temporary files.

Answer 2

Great recommendations @dijkgraaf , especially polling too often is frequent issue with customer having lots of receive locations and never need files too often with this protocol. Most customers I meet happen to have frequent pollings, but they may only need files every 15 minutes/1 hour anyway. I have also seen quite a lot of customers who has heavy traffic and antivirus enabled on the temp locations, which is blocking deletion of the files.

Sometimes only the dir listing remotely can take many seconds if there are many files on the busy remote server with higher latency/bandwidth. Make sure to adjust your polling interval to higher value if that is the case. You don't want to poll again before the previous dir listing is completed. To avoid long dir listings of files you don't need, try to use subfolders and get all * . * files instead of filtering in a big folder with lots of different files.

It can also be good to have a bit of random pollings. E g if you have 100 SFTP receive location, spread them into few different host instances and use different polling times so not all timer events, temp file writings etc happen at exactly the same time, e g set 151, 157, 163, 167, 181, 189, 191, 197 second polling seconds in different locations.

If you have multiple BizTalk servers in the group, make sure to only run non-transactional FTP, SFTP, POP3 on one host instance at a time. You can cluster the receive host to maintain high availability.