Jupyterlab "Too many open files" cause by duplicate loads of libraries

40 Views Asked by At

I repeatedly get jupyter lab sessions that start to hang while spamming tracebacks to the terminal that end with:

    zmq.error.ZMQError: Too many open files

An example complete traceback is:

[E 2024-03-17 17:40:36.843 ServerApp] Uncaught exception GET /api/kernels/059a770b-436f-40b5-b41b-95a1748ef7fd/channels?session_id=f60b1486-f926-4c96-924d-692a4e50d1e4 (172.16.206.48)
    HTTPServerRequest(protocol='http', host='localhost:23456', method='GET', uri='/api/kernels/059a770b-436f-40b5-b41b-95a1748ef7fd/channels?session_id=f60b1486-f926-4c96-924d-692a4e50d1e4', version='HTTP/1.1', remote_ip='172.16.206.48')
    Traceback (most recent call last):
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/tornado/websocket.py", line 944, in _accept_connection
        await open_result
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/websocket.py", line 77, in open
        await self.connection.connect()
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/connection/channels.py", line 363, in connect
        self.create_stream()
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/connection/channels.py", line 154, in create_stream
        self.channels[channel] = stream = meth(identity=identity)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/ioloop/manager.py", line 25, in wrapped
        socket = f(self, *args, **kwargs)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/connect.py", line 664, in connect_iopub
        sock = self._create_connected_socket("iopub", identity=identity)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/connect.py", line 654, in _create_connected_socket
        sock = self.context.socket(socket_type)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/zmq/sugar/context.py", line 362, in socket
        s: ST = socket_class(  # set PYTHONTRACEMALLOC=2 to get the calling frame
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/zmq/sugar/socket.py", line 159, in __init__
        super().__init__(
      File "zmq/backend/cython/socket.pyx", line 332, in zmq.backend.cython.socket.Socket.__init__
    zmq.error.ZMQError: Too many open files

This lab session is only running a single notebook. While that notebook is large, it's not extremely large.

I've investigated some, and lsof reveals that there are >30 copies of various library files being opened. For example:

$ lsof | grep scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so | wc
     33     361    8613
$ lsof | grep indexing.cpython-310-x86_64-linux-gnu.so | wc
     33     361    8481

These individually look like this:

COMMAND       PID     TID TASKCMD             USER   FD      TYPE             DEVICE    SIZE/OFF               NODE NAME
python    1325008                     adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325015 ZMQbg/Rea   adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325016 ZMQbg/IO/   adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325017 python      adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325018 python      adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so

i.e., they are all loaded by the same PID but different TIDs.

So, my question: What would cause a jupyter python session to load 30 copies of the libraries, and how do I prevent it?

0

There are 0 best solutions below