I am trying to implement custom model training on Vertex AI.
Every time I pushed custom training job to Vertex AI, I could meet the freezing issue in the middle of the process.
The issue occurs in,
message: "Generating new fontManager, this may take some time..."
pathname: "/opt/conda/lib/python3.7/site-packages/matplotlib/font_manager.py"
I tried to remove matplotlib package either, but it still tries to setup fontManager during the process.
Is there anyone who've been facing a same issue or have solved before? Thanks,
I tried to remove all the importing part of matplotlib in the scripts, but it still seems like vertex AI training job tries to install it.