IPython-Parallel: ipcluster and ipcontroller-client.json not found

Question

I'm trying to follow these instructions on using ipyparallel in order to speed up some Python scripts in Jupyter Notebook. When doing

import ipyparallel as ipp
IPPC = ipp.Client()

I get the following error:

Using matplotlib backend: TkAgg

Waiting for connection file: ~/.ipython/profile_default/security/ipcontroller-client.json

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-11-bf8e572b63bc> in <module>()
     18 display(HTML("<style>.container {width:100% !important;}</style>"))
     19 
---> 20 IPPC = ipp.Client()
     21 
     22 

/home/guest/.local/lib/python3.7/site-packages/ipyparallel/client/client.py in __init__(self, url_file, profile, profile_dir, ipython_dir, context, debug, sshserver, sshkey, password, paramiko, timeout, cluster_id, **extra_args)
    415                         no_file_msg,
    416                     ])
--> 417                     raise IOError(msg)
    418         if url_file is None:
    419             raise IOError(no_file_msg)

OSError: Connection file '~/.ipython/profile_default/security/ipcontroller-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.

The file ~/.ipython/profile_default/security/ipcontroller-client.json does not exist (folder is empty!!) and doing sudo find / -name ipcontroller-client doesn't return any results.

Furthermore, when trying to start ipcluster in the terminal, I get

user@laptop:~$ ipcluster start -n 2
ipcluster: command not found

and the "Clusters" tab in Jupyter Notebook is empty: Screenshot

However, ipyparallel is definitely installed:

user@laptop:~$ pip3 install ipyparallel
Requirement already satisfied: ipyparallel in ./.local/lib/python3.7/site-packages (6.3.0)
Requirement already satisfied: jupyter-client in /usr/lib/python3/dist-packages (from ipyparallel) (5.2.3)
Requirement already satisfied: traitlets>=4.3 in /usr/lib/python3/dist-packages (from ipyparallel) (4.3.2)
Requirement already satisfied: tornado>=4 in /usr/lib/python3/dist-packages (from ipyparallel) (5.1.1)
Requirement already satisfied: pyzmq>=13 in /usr/lib/python3/dist-packages (from ipyparallel) (17.1.2)
Requirement already satisfied: ipykernel>=4.4 in /usr/lib/python3/dist-packages (from ipyparallel) (4.9.0)
Requirement already satisfied: ipython-genutils in /usr/lib/python3/dist-packages (from ipyparallel) (0.2.0)
Requirement already satisfied: decorator in /usr/lib/python3/dist-packages (from ipyparallel) (4.3.0)
Requirement already satisfied: ipython>=4 in /usr/lib/python3/dist-packages (from ipyparallel) (5.8.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/lib/python3/dist-packages (from ipyparallel) (2.7.3)
Requirement already satisfied: pexpect in /usr/lib/python3/dist-packages (from ipython>=4->ipyparallel) (4.6.0)

and there are three file ipcluster, ipcontroller, and ipengine in /home/user/.local/bin which are Python scripts.

...what could be the problem?

Answer 1

Could it be that you didn't enable the clusters extension?

After installing ipyparallel you need to enable the notebook extension to be able to start clusters from the Jupyter notebook

ipcluster nbextension enable

You might need to re-install ipyparallel before:

pip uninstall notebook ipyparallel
pip install notebook ipyparallel
ipcluster nbextension enable

(the ipyparallel extension can also be enabled from the notebook's nbextensions_configurator if there's one installed)

Answer 2

I had to run ipcluster nbextension enable --user Otherwise, I got the error PermissionError: [Errno 13] Permission denied: '/usr/local/etc/jupyter'