Leaking file handles in call from flask -> dramatiq -> eventlet

26 Views Asked by At

Since I switched my python backend to eventlet I'm starting to run into "Too many open files" issues.

I start my process with gunicorn --worker-class eventlet -w 1 app:app

There are several code points which trigger this, the one used which causes the exception below:

@app.route('/ping_process', methods=['GET'])
def ping_process():
    process.ping_process.send()
  • This code block is called every 3 minutes by an uptime monitor.
  • process.ping_process is a dramatiq worker
  • after a few hours the python process uses more than the allowed 1024 file handles and causes the stack trace below
  • since this error started occurring when I switched to eventlet I'm pretty sure it's caused by eventlet. I'm using version 0.33.3

Anyone knows how to debug this?

Traceback (most recent call last):
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/philipp/backl-backend/app.py", line 103, in ping_process
    process.ping_process.send()
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/dramatiq/actor.py", line 137, in send
    return self.send_with_options(args=args, kwargs=kwargs)
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/dramatiq/actor.py", line 162, in send_with_options
    return self.broker.enqueue(message, delay=delay)
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/dramatiq/brokers/rabbitmq.py", line 329, in enqueue
    self.channel.basic_publish(
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/dramatiq/brokers/rabbitmq.py", line 163, in channel
    channel = self.state.channel = self.connection.channel()
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/dramatiq/brokers/rabbitmq.py", line 135, in connection
    connection = self.state.connection = pika.BlockingConnection(
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 435, in _create_connection
    ioloop = select_connection.IOLoop()
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/adapters/select_connection.py", line 374, in __init__
    self._poller = self._get_poller(self._get_remaining_interval,
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/adapters/select_connection.py", line 431, in _get_poller
    poller = SelectPoller(**kwargs)
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/adapters/select_connection.py", line 626, in __init__
    self._r_interrupt, self._w_interrupt = self._get_interrupt_pair()
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/adapters/select_connection.py", line 916, in _get_interrupt_pair
    return pika.compat._nonblocking_socketpair()  # pylint: disable=W0212
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/pika/compat.py", line 240, in _nonblocking_socketpair
    lsock = socket.socket(family, socket_type, proto)
  File "/home/philipp/backl-backend/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 139, in __init__
    fd = _original_socket(family, *args, **kwargs)
  File "/usr/lib/python3.10/socket.py", line 232, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
0

There are 0 best solutions below