AWS Lambda: IOException: Too many open files

Question

I deployed an AWS Lambda written in C#, and depending on the input I can see in the console the following error message (truncated):

{
  "Cause": "{\n  \"errorType\": \"IOException\",\n  \"errorMessage\": 
\"Too many open files : [...]at System.IO.FileSystem.RemoveDirectoryRecursive(String fullPath)
 "Error": "IOException",
  "ExecutionArn": [...]

I have a (somewhat?¹) good sense of what the error refer to, but I would like to reproduce it at least locally.

When I execute it using AWS .NET Lambda test tool with the exact same input (JSON), I cannot trigger the exception. So I assume that my Windows 10 system is more forgiving about the number of file descriptors than the one on AWS side. So added a call to _setmaxstdio in my Lambda code:

[DllImport("msvcrt.dll")]
public static extern int _setmaxstdio(int newMax);

And then in the main code I call:

int ret = _setmaxstdio(128);

I see a value of 128 being returned in the debugger, but again I cannot reproduce the issue locally using AWS .NET Lambda test tool. I picked 128 so that it is lower than the 1024 value described at:

• https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution

How would one reproduce the exception locally ?

---

^{¹ I suspect that Task.WhenAll is being called on all 1024 threads, which is exactly reaching the limit of 1024 open file descriptors.}

Answer 1

I am not sure if this will be more helpful, but the AWS Serverless Application Model (SAM) will let you run your C# lambda code locally against AWS resources. It also helps with deployment and additional infrastructure like S3 buckets or DyanmoDB tables. This blog is a good intro to the subject.

Answer 2

File handle limits on Windows don't work the same way as fd limits on Linux. Try running your dotnet code in a Linux machine or container locally to reproduce the issue in a more representative environment. You can artificially set low limits using "ulimit -f 128", for example. You can find out more about Microsoft's official .NET container images and how to get started here: https://hub.docker.com/_/microsoft-dotnet-samples/

That said, are you sure you're closing all your file handles appropriately? Anything that implements IDisposable should be used within a "using" statement or block to avoid leaking file handles and creeping toward that limit.

https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/statements/using