I am evaluating portia and run in to an issue deploying to scrapyd.
When I try to deploy my portia project using
scrapyd-deploy local -p new_project
from my portia project directory I get the following error message
Packing version 1433441798
Deploying to project "new_project" in http://192.168.59.103:6800/addversion.json
Server response (200):
{"status": "error", "message": "ImportError: Error loading object
'slybot.spidermanager.ZipfileSlybotSpiderManager': No module named slybot.spidermanager"}
I have one docker container running portia and another running scrapyd.
My project runs via portiacrawl
successfully.
I also tried creating a simple scrapy project using the startproject
and genspider
commands which deployed successfully on scrapyd
I'm not sure where to find the egg file that is generated.
Scrapyd log
2015-06-04 18:28:51+0000 [HTTPChannel,21,172.17.42.1] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 1618, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 773, in requestReceived
self.process()
File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 132, in process
self.render(resrc)
File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 167, in render
body = resrc.render(self)
--- <exception caught here> ---
File "/usr/lib/pymodules/python2.7/scrapyd/webservice.py", line 18, in render
return JsonResource.render(self, txrequest)
File "/usr/lib/pymodules/python2.7/scrapy/utils/txweb.py", line 10, in render
r = resource.Resource.render(self, txrequest)
File "/usr/lib/python2.7/dist-packages/twisted/web/resource.py", line 216, in render
return m(request)
File "/usr/lib/pymodules/python2.7/scrapyd/webservice.py", line 66, in render_POST
spiders = get_spider_list(project)
File "/usr/lib/pymodules/python2.7/scrapyd/utils.py", line 65, in get_spider_list
raise RuntimeError(msg.splitlines()[-1])
exceptions.RuntimeError: ImportError: Error loading object 'slybot.spidermanager.ZipfileSlybotSpiderManager': No module named slybot.spidermanager
2015-06-04 18:28:51+0000 [HTTPChannel,21,172.17.42.1] 172.17.42.1 - - [04/Jun/2015:18:28:51 +0000] "POST /addversion.json HTTP/1.1" 200 156 "-" "Python-urllib/2.7"
Any ideas what I am missing ?
I was able to basically resolve this by using the latest from github instead of 1.0.1 version and a scrapy version of 0.25.1
More info here: https://github.com/scrapinghub/portia/issues/209