Can't run IPython-Parallel in Jupyter Notebook

Question

I'm trying to follow these instructions on using ipyparallel in order to speed up some Python scripts in Jupyter Notebook. When doing

import ipyparallel as ipp
IPPC = ipp.Client()

I get the following error:

Using matplotlib backend: TkAgg

Waiting for connection file: ~/.ipython/profile_default/security/ipcontroller-client.json

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-11-bf8e572b63bc> in <module>()
     18 display(HTML("<style>.container {width:100% !important;}</style>"))
     19 
---> 20 IPPC = ipp.Client()
     21 
     22 

/home/guest/.local/lib/python3.7/site-packages/ipyparallel/client/client.py in __init__(self, url_file, profile, profile_dir, ipython_dir, context, debug, sshserver, sshkey, password, paramiko, timeout, cluster_id, **extra_args)
    415                         no_file_msg,
    416                     ])
--> 417                     raise IOError(msg)
    418         if url_file is None:
    419             raise IOError(no_file_msg)

OSError: Connection file '~/.ipython/profile_default/security/ipcontroller-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.

The file ~/.ipython/profile_default/security/ipcontroller-client.json does not exist (folder is empty!!) and doing sudo find / -name ipcontroller-client doesn't return any results.

Furthermore, when trying to start ipcluster in the terminal, I get

user@laptop:~$ ipcluster start -n 2
ipcluster: command not found

and the "Clusters" tab in Jupyter Notebook is empty: Screenshot

However, ipyparallel is definitely installed:

user@laptop:~$ pip3 install ipyparallel
Requirement already satisfied: ipyparallel in ./.local/lib/python3.7/site-packages (6.3.0)
Requirement already satisfied: jupyter-client in /usr/lib/python3/dist-packages (from ipyparallel) (5.2.3)
Requirement already satisfied: traitlets>=4.3 in /usr/lib/python3/dist-packages (from ipyparallel) (4.3.2)
Requirement already satisfied: tornado>=4 in /usr/lib/python3/dist-packages (from ipyparallel) (5.1.1)
Requirement already satisfied: pyzmq>=13 in /usr/lib/python3/dist-packages (from ipyparallel) (17.1.2)
Requirement already satisfied: ipykernel>=4.4 in /usr/lib/python3/dist-packages (from ipyparallel) (4.9.0)
Requirement already satisfied: ipython-genutils in /usr/lib/python3/dist-packages (from ipyparallel) (0.2.0)
Requirement already satisfied: decorator in /usr/lib/python3/dist-packages (from ipyparallel) (4.3.0)
Requirement already satisfied: ipython>=4 in /usr/lib/python3/dist-packages (from ipyparallel) (5.8.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/lib/python3/dist-packages (from ipyparallel) (2.7.3)
Requirement already satisfied: pexpect in /usr/lib/python3/dist-packages (from ipython>=4->ipyparallel) (4.6.0)

and there are three file ipcluster, ipcontroller, and ipengine in /home/user/.local/bin which are Python scripts.

What could be the problem?

Answer 1

Sarah.Connr, I recently had a similar initial experience with ipyparallel, where the symptoms were much the same. After installing it, the folders, files, and Jupyter Notebook "clusters" tab contents you mention were missing.
The issue was that although it looked like ipyparallel was installed, it was either installed incorrectly, or configuration, permissions or owners (user vs. root) were messed up.

But first, you reported:

user@laptop:~$ ipcluster start -n 2  
ipcluster: command not found  

and "there are three files: ipcluster, ipcontroller, and ipengine in /home/user/.local/bin which are Python scripts."

On my ubuntu system, those programs installed in /usr/local/bin, not ~/.local/bin.
In any case, ipcluster: command not found means that ipcluster is not in your execution path.
You can run it by entering the full path to the program or first cd-ing to its folder.
This might solve your problem, or it might reveal more issues.

OK, now for my story and its solution...
I first installed ipyparallel:

$ pip3 install ipyparallel

There were problems, so subsequently,

$ pip3 install --user ipyparallel

Problems remained, so,

$ sudo pip3 install ipyparallel

I saw the "Requirement already satisfied:..." messages, too, but they were preceded by this warning:

WARNING: The directory '/home/(user name here)/.cache/pip' or its parent  
directory is not owned or is not writable by the current user. The cache  
has been disabled. Check the permissions and owner of that directory.  
If executing pip with sudo, you may want sudo's -H flag.

Things still didn't work!
So I ran

$ sudo -H pip3 install ipyparallel

At this point, all the folders and files showed up, the appropriate items appeared under the Jupyter Notebook "clusters" tab, and the problems disappeared.

Now I'll admit that I don't understand all of this, and I do tend to tinker by instinct and experience until things eventually work (or not). So I'm not saying that you need to (or should) do this, but I hope my comments provide a starting point and some helpful hints.