Scrapy dmoz tutorial: _init_() takes at most 2 arguments (3 given)

480 Views Asked by At

PS C:\users\steve\tutorial> scrapy crawl dmoz

Traceback (most recent call last):

File "c:\python27\scripts\scrapy-script.py", line 9, in <module> 
  load_entry_point('scrapy==1.0.3', 'console_scripts', 'scrapy')()
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\cmdline.py",
  cmd.crawler_process = CrawlerProcess(settings)
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\crawler.py",
  super(CrawlerProcess, self).__init__(settings)
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\crawler.py",
  self.spider_loader = _get_spider_loader(settings)
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\crawler.py",
  return loader_cls.from_settings(settings.frozencopy())
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\spiderloader.
  return cls(settings)
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\spiderloader.
  for module in walk_modules(name):
File "C:\Python27\lib\site-packages\scrapy-1.0.3-py2.7.egg\scrapy\utils\misc.py
  submod = import_module(fullpath)
File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
  __import__(name)
File "C:\users\steve\tutorial\tutorial\spiders\dmoz.py", line 4, in <module>
  class dmozspider(spiders):
TypeError: Error when calling the metaclass bases module.__init__() takes at most 2 arguments (3 given)

My dmoz spider python script is here

from scrapy import spiders


class dmozspider(spiders):  
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
     ]

    def parse(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)
1

There are 1 best solutions below

1
On BEST ANSWER

The problem is you're importing "spiders", and using it as your base class. "spiders" is the package that contains the spiders, namely the Spider class. To use it, use:

from scrapy.spiders import Spider


class dmozspider(Spider):
    ...  # Rest of your code