I have an IPython notebook that I was running as multi-threaded and have changed over to try and run on multiple processors. There is a substantial amount of code that I won't quote here unless someone thinks it would be helpful to look at a specific portion of it. The gist of it is that I'm trying to wrap a load balancer around PyOpenCL so I need to feed multiple independent queues-- partly to solve a specific problem (particle simulation) and partly as a learning exercise.
At no point am I intentionally opening any files, yet it's crashing with this error:
Traceback (most recent call last):
File "//anaconda/lib/python3.4/site-packages/IPython/core/interactiveshell.py", line 3035, in run_code
File "<ipython-input-23-d29c87aba720>", line 1, in <module>
co.convergeStep(mags[3])
File "<ipython-input-17-0eac43f0bbbf>", line 62, in convergeStep
evt=self.nlb.qTask(evtList,(b,k),b.forces,k,out=b.forceAccum,out2=k.forceAccum,sum=True)
File "<ipython-input-13-0f26b8ae660e>", line 36, in qTask
t=StagingThread(waitlist,c,self.pool)
File "<ipython-input-10-549b980aea87>", line 7, in __init__
self.killme=mp.Event()
File "//anaconda/lib/python3.4/multiprocessing/context.py", line 91, in Event
File "//anaconda/lib/python3.4/multiprocessing/synchronize.py", line 336, in __init__
File "//anaconda/lib/python3.4/multiprocessing/context.py", line 76, in Condition
File "//anaconda/lib/python3.4/multiprocessing/synchronize.py", line 215, in __init__
File "//anaconda/lib/python3.4/multiprocessing/context.py", line 81, in Semaphore
File "//anaconda/lib/python3.4/multiprocessing/synchronize.py", line 127, in __init__
File "//anaconda/lib/python3.4/multiprocessing/synchronize.py", line 60, in
__init__ OSError: [Errno 24] Too many open files
Is there something opening files in the background that I'm not considering?
It looks like I have 11 processes spawned at the time of failure. I'm running under OS X 10.10.3, Python 3.4.3, IPython 3.1. I am not using multiprocessing.Pool (at least not intentionally). It looks like the failure is happening in Event? There are other exceptions that occur while handling this one, but they're mostly too many files errors and then some traceback failures and other things that look like they're a result of one of the above...