I'm running into an issue using pathos multiprocessing that runs figure saving function.
from pathos.multiprocessing import ProcessingPool as Pool
datasets = Pool().map(model_handler, analysisParams)
which throws
File ".../lib/python3.6/site-packages/matplotlib/backends/backend_pdf.py", line 2029, in draw_text
font.set_text(s, 0.0, flags=LOAD_NO_HINTING)
RuntimeError: In set_text: could not load glyph
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "wrapper.py", line 410, in <module>
datasets = Pool().map(model_handler, analysisParams)
File ".../lib/python3.6/site-packages/pathos/multiprocessing.py", line 137, in map
return _pool.map(star(f), zip(*args)) # chunksize
File ".../lib/python3.6/site-packages/multiprocess/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File ".../lib/python3.6/site-packages/multiprocess/pool.py", line 608, in get
raise self._value
RuntimeError: In set_text: could not load glyph
"""
My current understanding is that this occurs because glyph is being called simultaneously by multiple processes. The exception only happens when the number of parallel executions gets >3, which is consistent with that idea. I see that one person's work around was simply to call savefig repeatedly until the command went through, but the parallelized function contains a ton of plots and I'd rather not need to wrap each one in a try statement. Does anyone have an idea about how avoid this exception when plotting figures in parallel? Thanks!!
EDIT Minimal example per comment request.
from pathos.multiprocessing import ProcessingPool as Pool
def model_handler(analysisParam):
import matplotlib
matplotlib.use('agg')
import matplotlib.pyplot as plt
figout = plt.figure(figsize=(1, 1))
axes = figout.add_subplot(1,1,1)
axes.scatter(range(1000), range(1000))
figout.savefig('dummyfig{}.pdf'.format(analysisParam), format='pdf')
return analysisParam
datasets_serial = []
for j in range(20):
datasets_serial.append(model_handler(j))
datasets = Pool(20).map(model_handler, range(20))
Interestingly, if I replace the dataset_serial loop with dataset_serial = map(model_handler, range(20))
, the pathos parallel pool executes with no problem, but if I convert the serial map object to a list (print(list(dataset_serial))
), the glyph exception in the parallel pool comes back.