"Too many open files" during rspec tests on MacOS

Question

I am working on an open source Ruby on Rails project with a full suite of rspec tests. The tests work normally on the CI/CD pipeline server, but locally on my MacBook running Monterey they fail intermittently. Sometimes zero tests will fail, sometimes a handful, sometimes hundreds — all with no code changes.

The error messages

The intermittently failing tests produce a variety of errors:

Errno::EMFILE:
       Failed to open TCP connection to 127.0.0.1:9520 (Too many open files - socket(2) for "127.0.0.1" port 9520)

The above error appears with various port numbers. I have spotted 9515, 9517, 9518, 9520, 9522, 9529, 9533 and 9536.

I'm also seeing a lot of other errors that complain of too many open files:

Failure/Error: Capybara::Webmock.start Errno::EMFILE: Too many open files

Errno::EMFILE: Too many open files - rackup

Errno::EMFILE: Too many open files - /Users/username/.webdrivers/chromedriver

I think it likely that a single issue underlies all these intermittent errors. Search results for a lot of these errors lead to Chromedriver, Selenium, Capybara, and WebMock.

What I have tried

• I have increased the number of available file descriptors with ulimit -Sn 61440
• Increased the files available to launchctl limit maxfiles
• I have checked that my chromedriver -v corresponds to the Chrome browser version installed on my MacBook
• I have cleared caches by running bundle exec rails tmp:clear and Rails.cache.clear, and by setting the Cache-Control header to public, no-cache, must-revalidate and other settings
• I have added net_http_connect_on_start: true to my WebMock configuration, per their documentation
• I have tried killall chromedriver to ensure there isn't a zombie chromedriver process

My code

My WebMock configuration should allow local connections as it includes the following:

require 'webmock/rspec'
WebMock.disable_net_connect!(
  allow: [
    /localhost/,
    /127\.0\.0\.1/,
    /codeclimate.com/, # For uploading coverage reports
    /chromedriver\.storage\.googleapis\.com/, # For fetching a chromedriver binary
  ],
)

My Capybara configuration is here.

Answer 1

To answer my own question, I increased the available macOS file descriptors to solve this problem. MacOS apparently does not take you seriously when you increase the ulimit from the command line; it demands a configuration file and a restart.

I'm not convinced I found the underlying issue — I suspect there may be some problem in my configuration causing file descriptors to be exhausted. Perhaps there are some connections that aren't being property closed. Yet, increasing the limit solves the immediate problem.

I created a property list file owned by root:

sudo nano /Library/LaunchDaemons/limit.maxfiles.plist

I paste this text into the file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
          "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>maxfiles</string>
      <string>524288</string>
      <string>524288</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

I saved the file. Just to be sure, I printed the file to the terminal with cat to confirm it exists. And I checked ls to ensure that root wheel owned the file.

cat /Library/LaunchDaemons/limit.maxfiles.plist
ls -la /Library/LaunchDaemons/limit.maxfiles.plist

I restarted my MacBook to cause the .plist to take effect. Then I checked the file descriptor limits again with:

• ulimit -n
• launchctl limit maxfiles

Both now return a limit of 524288. The "too many open files" messages are gone.

Further reading:

1. El Capitan ulimit shenanigans by Dejan Kitic
2. "Too many open files" limit/ulimit on Mac OS X by Thomas Jung
3. Ruby 2.7.1 - Errno::EMFILE: Failed to open TCP connection to 127.0.0.1:9*** (Too many open files - socket(2) for "127.0.0.1" port 9***) by ndbroadbent