Knox process consumes all resources in Dataproc master node

Question

I try to open a my notebook that is stored in GCS. I create a Dataproc cluster with Jupyter and I connect Jupyter web interface.

When I open my notebook, I get an error(Error 504, there is not more info).

When I open my notebook, I check the master node and I see there is 'knox' process consumes my cpu(%198) and memory(%100) resources.

I encountered this situation 3-4 times every week.

Is there anyone who has encountered same issue before?

Answer 1

I reproduced your problem by creating a cluster with Jupyter component, then opened a Python notebook and ran print('hello world') many times. Then I sshed into the master node and kept monitoring the Knox process memory usage. Each time I ran the cell, the memory of Knox process increased by ~3M.

Setting export KNOX_GATEWAY_MEM_OPTS="-Xms512m -Xmx1024m" in /usr/lib/knox/bin/knox-env.sh then sudo systemctl restart knox might be helpful, but it doesn't seem to be the cause of your problem.