It is observed that the GluonTS deep state time series model is performing differently in different operating systems. The prediction takes around 3 seconds when it is run over docker container on Mac OS (Quad core, Physical VM, intel i7 @2.3 GHz); this function takes around 8 seconds when it is run over docker container on Linux OS (4 CPU only, VM, intel Broadwell @2.19 GHz); and the same function takes around 20 seconds when it is run over kubernetes cluster (allocated with 16 CPUs, VM, BM.standard2.52). When monitored the CPU / memory during the time, it is observed that the program is trying to consume all the allocated CPUs while minimally taking the memory and page files. I Ran cProfile on the Linux environment and below is the short output from the profiler. I understand that mxnet comes with different libraries to support the different OS. Is there any known issue around the mxnet performance? How best I can nail down the performance issue.
1 0.000 0.000 7.819 7.819 /usr/local/lib/python3.7/site-packages/xxxxxx/forecast/forecast_client.py:26(predict)
1 0.000 0.000 7.801 7.801 /usr/local/lib/python3.7/site-packages/xxxxxx/forecaster.py:8(predict)
2 0.000 0.000 7.801 3.900 /usr/local/lib/python3.7/site-packages/gluonts/mx/model/predictor.py:157(predict)
2 0.000 0.000 7.800 3.900 /usr/local/lib/python3.7/site-packages/gluonts/model/forecast_generator.py:113(__call__)
1 0.000 0.000 7.786 7.786 /usr/local/lib/python3.7/functools.py:835(wrapper)
1 0.000 0.000 7.786 7.786 /usr/local/lib/python3.7/site-packages/gluonts/mx/model/predictor.py:61(_)
5901/1 0.029 0.000 7.785 7.785 /usr/local/lib/python3.7/site-packages/mxnet/gluon/block.py:688(__call__)
5901/1 0.075 0.000 7.785 7.785 /usr/local/lib/python3.7/site-packages/mxnet/gluon/block.py:1127(forward)
1 0.001 0.001 7.764 7.764 /usr/local/lib/python3.7/site-packages/gluonts/model/deepstate/_network.py:216(hybrid_forward)
89852 3.323 0.000 4.976 0.000 /usr/local/lib/python3.7/site-packages/mxnet/_ctypes/ndarray.py:80(_imperative_invoke)
2 0.000 0.000 4.301 2.151 /usr/local/lib/python3.7/site-packages/gluonts/model/deepstate/_network.py:102(compute_lds)
2 0.002 0.001 4.239 2.119 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:803(unroll)
4 0.010 0.003 4.226 1.057 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:195(unroll)
5880/2352 0.021 0.000 4.162 0.002 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:287(forward)
2352 0.019 0.000 3.939 0.002 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:958(hybrid_forward)
1 0.001 0.001 2.847 2.847 /usr/local/lib/python3.7/site-packages/gluonts/mx/distribution/lds.py:201(log_prob)
1 0.024 0.024 2.840 2.840 /usr/local/lib/python3.7/site-packages/gluonts/mx/distribution/lds.py:238(kalman_filter)
2352 0.085 0.000 2.313 0.001 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:527(hybrid_forward)
672 0.041 0.000 2.290 0.003 /usr/local/lib/python3.7/site-packages/gluonts/mx/distribution/lds.py:581(kalman_filter_step)
1176 0.007 0.000 1.383 0.001 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:988(hybrid_forward)
2352 0.019 0.000 1.092 0.000 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:969(<listcomp>)
672 0.026 0.000 0.790 0.001 /usr/local/lib/python3.7/site-packages/gluonts/mx/distribution/multivariate_gaussian.py:73(log_prob)
89852 0.403 0.000 0.760 0.000 /usr/local/lib/python3.7/site-packages/mxnet/ndarray/register.py:75(_verify_all_legacy_ndarrays)
4704 0.020 0.000 0.740 0.000 /usr/local/lib/python3.7/site-packages/mxnet/gluon/rnn/rnn_cell.py:961(<lambda>)
10249 0.052 0.000 0.666 0.000 <string>:2(linalg_gemm2)
About environment:
- python 3.7
- gluonts 0.6.1
- mxnet 1.6.0