ckan harvester: "No module named pika" error

Question

On a ckan instance running ok, I installed the harvester extension following this guide: https://github.com/ckan/ckanext-harvest

these are the steps I followed:

. /usr/lib/ckan/default/bin/activate
cd /usr/lib/ckan/default/src/ckan

sudo pip install -e git+https://github.com/okfn/ckanext-harvest.git@stable#egg=ckanext-harvest

cd /usr/lib/ckan/default/src/ckan/src/ckanext-harvest
sudo pip install -r pip-requirements.txt

This is the content of pip-requirements.txt:

pika==0.9.8
redis==2.10.1

I continue configuring the plugin, everything seems to work ok. I have it running at http://localhost/harvest. Then I create a new source, and when I want to start the gather command I get this error:

$ . /usr/lib/ckan/default/bin/activate
$ cd /usr/lib/ckan/default/src/ckan/src/ckanext-harvest
$ paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/default/production.ini 

Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 125, in command
    from ckanext.harvest.queue import get_gather_consumer, gather_callback
  File "/usr/lib/ckan/default/src/ckan/src/ckanext-harvest/ckanext/harvest/queue.py", line 5, in <module>
    import pika
ImportError: No module named pika

I'm pretty sure there must be something really silly with the virtualenv (python newbie here)

Answer 1

This is because you used sudo pip. Because of how python's virtualenv works, if you want to use sudo to install into the virtualenv, you need to give full path to pip. Something like this would work

sudo /usr/lib/ckan/default/bin/pip -e git+https://github.com/okfn/ckanext-harvest.git@stable#egg=ckanext-harvest
cd /usr/lib/ckan/default/src/ckanext-harvest
sudo /usr/lib/ckan/default/bin/pip install -r pip-requirements.txt